gh-136757: Reuse static strings for exact operator tokens

[omkar-334]

This PR reduces repeated string allocations in the tokenizer by reusing static interned strings for exact multi-character operator tokens such as ==, ->, //=, and ....

Before -

"==" repeated 1,000 times -> 1,000 distinct "==" string objects

After -

"==" repeated 1,000 times -> 1 distinct "==" string object

• Issue: Intern string representation of operators and some other symbolic literals #136757

Benchmarks

These are local synthetic measurements from pyperf --rigorous on a quieter machine. The public tokenize timing cases are the strongest timing evidence; one direct _tokenize.TokenizerIter streaming case still had a pyperf stability warning.

Workload:

• 2,000 repetitions of a source snippet containing all 24 candidate multi-character exact operators.
• Compared public tokenize.tokenize() and private _tokenize.TokenizerIter.
• Measured both streaming token consumption and retained token streams.

Object reuse:

Repeated "==" tokens:
before: 1,000 distinct str objects
after:      1 distinct str object

Retained operator token strings:
before: 48,000 distinct str objects for 24 values
after:      24 distinct str objects for 24 values

Peak memory for retained token streams:

list(tokenize.tokenize):
before: 70.091 MiB
after:  68.113 MiB
change: -2.82%

list(_tokenize.TokenizerIter):
before: 65.054 MiB
after:  63.076 MiB
change: -3.04%

Timing:

tokenize.tokenize count:
51.0 ms +- 0.7 ms -> 48.5 ms +- 0.8 ms
1.05x faster

list(tokenize.tokenize):
91.1 ms +- 1.6 ms -> 88.6 ms +- 1.1 ms
1.03x faster

_tokenize.TokenizerIter count:
28.6 ms +- 0.6 ms -> 26.8 ms +- 0.7 ms
1.07x faster
note: this case had a pyperf stability warning

list(_tokenize.TokenizerIter):
51.8 ms +- 1.0 ms -> 50.7 ms +- 0.6 ms
1.02x faster

ast.parse control:
no significant difference

[omkar-334]

cc @ZeroIntensity for review

[StanFromIreland]

Thanks for benchmarking, could you please also share the script you used?

2,000 repetitions of a source snippet containing all 24 candidate multi-character exact operators.

So that means that 1.05x is the upper bound for a synthetic corpus, but not what anyone tokenizing real code would see. Can you please also benchmark a realistic corpus (e.g., a file from the stdlib)?

list(_tokenize.TokenizerIter):
51.8 ms +- 1.0 ms -> 50.7 ms +- 0.6 ms
1.02x faster

This is noise, the delta is below the combined error and the ranges overlap.

The change is small, but so is the win. I'm -0 on this change, but I won't object if someone else thinks it's worth it.

[ZeroIntensity]

This test passes on main and isn't related to this change; please remove it.

[ZeroIntensity]

I'm not a fan of getting _tokenize this way, nor do I think this test stresses anything different from the first one. Let's keep just the first test and remove this one.

[ZeroIntensity]

Use sys._is_interned instead of relying on is.