Python: fix PEP 758 except A, B: extraction in the default parser - #22386
Open
aausch wants to merge 2 commits into
Open
Python: fix PEP 758 except A, B: extraction in the default parser#22386aausch wants to merge 2 commits into
except A, B: extraction in the default parser#22386aausch wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes default Python parser extraction of PEP 758 exception lists, aligning it with the tree-sitter parser.
Changes:
- Distinguishes
asaliases from comma-separated exception types. - Adds parser parity regression coverage.
- Documents the corrected extraction behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
python/extractor/semmle/python/parser/ast.py |
Extracts except A, B: as a load-context tuple. |
python/extractor/tests/parser/exceptions_relaxed.py |
Tests exception-list and alias variants across parsers. |
python/ql/lib/change-notes/2026-08-19-legacy-parser-relaxed-except.md |
Records the parser fix and query impact. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The grammar rule shared by both readings is
except_clause: 'except' [test [(',' | 'as') test]]
and `visit_except_clause` ignored the separator token, always treating the
fourth child as an alias to bind. So `except A, B:` extracted `B` as a Store
rather than a use, which is the Python 2 reading. Queries that reason about
whether a name is used then report false positives; `py/unused-import` flags
the import of `B` as unused.
The tree-sitter parser already extracts this as a tuple of exception types
(github#20990), so the two parsers disagreed. `tests/parser/exceptions_relaxed.py`
is an unsuffixed parser test, which asserts the two parsers produce identical
ASTs; it fails without this change.
With the fix, the default parser reproduces the existing
`tests/parser/exceptions_new.expected` byte for byte, and of the 37 parser
test files only the two containing PEP 758 syntax change at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ngth The parser test added with the fix pins the AST. These pin the behaviour a user actually sees, through the real extraction path including the tree-sitter fallback, and they cover chains longer than two. Chains of three or more behave differently from chains of two, which is worth having written down. `except A, B, C:` fails the default parser outright, so `Module.py_ast` falls back to tree-sitter and the result is already correct. `except A, B:` parses successfully under the Python 2 reading, so the fallback never fires and the bad AST reaches the queries. That is why only the two-type form produced a false positive. It also means the two cases cannot share a file: any three-type clause sends the whole file to tree-sitter and masks the two-type behaviour. Hence relaxed_except.py and relaxed_except_long.py, with a comment in each saying so. Each name is used in exactly one clause for the same reason -- a name reused in a parenthesized clause is a use regardless, and hides the defect. Verified by reverting the extractor fix in a 2.26.3 bundle: relaxed_except.py then reports `Import of 'Beta' is not used.` and the test fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aausch
force-pushed
the
aausch/python-pep758-legacy-parser
branch
from
August 20, 2026 09:10
03c2f1e to
896d8d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
except A, B:(PEP 758, Python 3.14+) is extracted by the default parser using the Python 2 reading:Bbecomes an alias binding (Store) instead of a use (Load). Queries that reason about whether a name is used then misfire —py/unused-importreports the import ofBas unused.Why
blib2to3/Grammar.txtshares one rule between both readings:and
visit_except_clausenever looked at the separator, so a fourth child was always bound as an alias:This patch checks the separator token:
asbinds an alias as before,,builds a tuple of exception types withLoadcontext and no alias.Only the two-type form was affected.
except A, B, C:already extracted correctly, and parenthesized forms were never affected.Verification
python/extractor/tests/parser/exceptions_new.expected— the tree-sitter parser's expected AST for this exact syntax — byte for byte, locations and contexts included.tests/parser/before and after, only the two files containing PEP 758 syntax differ.tests/parser/exceptions_relaxed.pyis unsuffixed, so the harness asserts the two parsers produce identical ASTs. It fails onmainand passes with this change.pytest tests/test_parser.py→ 37 passed.codeql-bundle-v2.26.3extractor and re-runningImports/UnusedImport.qlover the reproduction removes the false positive, while a genuinely unused import in the same file is still reported.python/ql/test/query-tests/Imports/unused/gains coverage for two-, three- and four-type chains through the real extraction path. Reverting the extractor fix in a 2.26.3 bundle makesrelaxed_except.pyreportImport of 'Beta' is not used.and the test fail.Chains longer than two
Worth recording, because the two cases behave differently and the difference is not obvious.
except A, B:except A, B, C:SyntaxErrorModule.py_astfalls back to tree-sitter, AST is correctexcept A, B, C, D:SyntaxErrorexcept_clause: 'except' [test [(',' | 'as') test]]admits exactly one trailingtest, so three or more types cannot parse at all. The fallback in
semmle/python/modules.pythen rescues them. Only the two-type form is silentlywrong, precisely because it is the only one the default parser accepts.
This has a consequence for testing that cost me a first attempt: the two cases
cannot share a file. Any three-type clause fails the default parser, sends the
whole file to tree-sitter, and masks the two-type behaviour completely. So the
query tests are split into
relaxed_except.pyandrelaxed_except_long.py,with a comment in each explaining why. For the same reason each name is used in
exactly one clause — a name that also appears in a parenthesized clause is a use
regardless, and hides the defect.
I have deliberately not changed the grammar to accept longer chains. The
fallback already yields correct results, and a
Grammar.txtchange is a muchlarger and riskier diff than the defect warrants. It may still be worth doing:
the fallback costs a failed parse per affected file, which measured at roughly
541ms versus 9ms for a neighbouring file in the same run. Happy to open that
separately if you would like it.
Trade-off
Python 2 source of the form
except ValueError, e:that parses under the Python 3 grammar will now be read as a tuple of exception types rather than an alias binding. This input is genuinely ambiguous between the two language versions and the grammar rule cannot distinguish them. The tree-sitter parser (#20990) already resolves it in favour of PEP 758, and the unsuffixed parser tests require the two parsers to agree, so this change aligns them. Happy to gate it on the matched grammar instead if you would rather preserve the Python 2 reading — that needs the grammar identity threaded fromparser/__init__.py::parseintoast.convert, which I left out to keep this minimal.Fixes #22387
Investigated with assistance from Claude Code. Every result quoted above was executed against the real 2.26.3 bundle and the repo's own test harness, not inferred.