gh-151464: exclude '<>' token from tokenize output

[skirpichev]

Was:

$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

Now (regardless on __future__.barry_as_FLUFL import):

$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for <> ("not equal" in Pascal and Python 2).

• Issue: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

[Aniketsy]

Thanks for this work :)

I've just gone through these changes and changes looks good as per my understanding ofcourse i'm not expert 😀 , and it was really interesting to going through these changes, at first I found its bit tricky to understand some portion.

I'm excited to see the review process in this from experts and try to understand how it goes, also if you feel this comment as noise please feel free to mark as off-topic.

[pablogsal]

Left some comments

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[skirpichev]

Left some comments

I have made the requested changes; please review again

[bedevere-app]

Thanks for making the requested changes!

@pablogsal: please review the changes made to this pull request.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34005059 | 📁 Comparing 3f9c57e against main (219768f)

  🔍 Preview build  

2 files changed

± whatsnew/3.16.html
± whatsnew/changelog.html