Fix jcl and m4 func_start recall: route to Mode A, not brace search - #1987
Merged
Conversation
Same bug shape as dockerfile (#1976): jcl and m4 have no ScopeParsingRegistry entry and no brace-delimited bodies at all, so they silently fell through to Mode_B_Braces in _function_slice -- which only produces a named function when a literal { happens to appear by coincidence within its search window. jcl's func_start matches "// <name> EXEC ..." job-step lines -- JCL is fixed-column mainframe syntax with no brace concept anywhere. m4's func_start matches define(...)/m4_define(...)/AC_DEFUN(...) etc. -- macro definitions are parenthesis-delimited with backtick/bracket quoting, not brace-delimited. Confirmed via direct measurement (struct_func_start, the raw signal, vs. function_count, the named list actually reaching consumers): jcl: 3 raw matches, 0 reached the named list (0% recall) m4: 39 raw matches, 1 reached the named list (2.6% recall) Routes both to Mode A (_slice_by_labels), the same greedy-to-next- match heuristic already proven for abap/cobol/fortran/assembly/ dockerfile. Post-fix: both hit 100% recall in the local corpus (3/3, 39/39). makefile was also flagged as a candidate in the same investigation but its corpus sample was inconclusive (1 match total) -- deliberately left untouched, not part of this fix's scope. Verified against the full ~80-repo crucible corpus (both golden masters re-blessed; drift is real function-count/structural-mass changes in jcl/m4 files plus expected global spatial-layout ripple across the corpus, not corruption elsewhere). Fixes #1975. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-Authored-By: Gemini 3.1 Pro <noreply@google.com>
squid-protocol
enabled auto-merge (squash)
August 21, 2026 01:46
…ting-1975 # Conflicts: # tests/golden_master_audit.json # tests/golden_master_zero_dep_audit.json
Contributor
🐦⬛ Muninn Security Scan
ℹ️ Info Findings[checkov] Ensure that a user for the container has been createdFile: [checkov] Ensure that HEALTHCHECK instructions have been added to container imagesFile: [checkov] Ensure the base image uses a non latest version tagFile: |
squid-protocol
added a commit
that referenced
this pull request
Aug 21, 2026
…-shift Second conflict round: #1987 (jcl/m4) landed on main right after the first resolution, requiring another regenerate-and-rebless pass. Also fixes a real, separate issue caught while investigating why PR #1985/#1987 merged despite a failing ruff-audit CI check: #1972 (record_keeper.py's class_count fix, merged earlier and unrelated to this PR) shifted 7 pre-existing baselined findings by a couple of lines each -- tests/ruff_audit_baseline.json was never regenerated to absorb that shift, so every PR's ruff-audit CI run since has been reporting those 7 as spurious 'new' findings. Confirmed via audit_check.py that all 7 are pure line-shifts (same file/code/ message, just moved), not real regressions, and regenerated the baseline to absorb them. Verified against the exact CI-pinned ruff version (0.16.0, not whatever floats on PATH locally).
squid-protocol
added a commit
that referenced
this pull request
Aug 21, 2026
* Add named class (build-stage) extraction for dockerfile Dockerfile had no named class extraction at all: it wasn't in _CLASS_START_NAMED_EXTRACTION_LANGS, so it fell back to the legacy generic regex (class|struct|interface|trait|enum), which never matches Dockerfile's FROM syntax at all -- correctly, since Dockerfile has none of those keywords. Result: class_data stayed completely empty for every Dockerfile scanned, even though struct_class_start (the raw signal) correctly counted every real FROM line. class_start's own regex previously captured only the literal keyword FROM itself, not the build stage's real name -- extended it to an alternation shape (matching the existing Fortran/Lua/ABAP convention _resolve_class_start_match already documents): group 1 captures the AS <alias> name when present, group 2 falls back to the bare base-image reference for a stage with no alias (the file's final/default stage). --platform=$VAR-style flags between FROM and the image reference are skipped over. Added dockerfile to _CLASS_START_NAMED_EXTRACTION_LANGS, and to the boundary-resolution skip condition alongside abap (same #1907 rationale -- Dockerfile stages are never nested, and the brace search would mistake ${VAR} template-substitution braces for a real body opener). Verified: class_data row counts now match struct_class_start exactly (69/6/1/1 across the moby test corpus), with real stage names (base, criu, xx, binary-dummy, ...) instead of the literal string "FROM" repeated. ReDoS-probed the new regex directly (5000-char runs, deeply nested continuations, huge flag/image tokens) -- all sub-millisecond. One known, accepted side effect: THE LINEAGE EXTRACTOR (detector.py's generic "any class_start match with 2+ groups treats group 2 as an inheritance parent") doesn't know groups 1/2 here are alternation- exclusive, not name-then-parent -- a bare FROM <image> (no alias) sweeps the image reference into that file's parent_entity metadata. This is a pre-existing, already-shipped pattern (fortran's own class_start has the identical alternation shape and triggers the same behavior for bare TYPE declarations today), not something this fix introduces -- tracked as its own issue (#1983) for a future, more alternation-aware fix to the shared extractor, not blocking here. Fixes #1974. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Co-Authored-By: Gemini 3.1 Pro <noreply@google.com> * Regenerate golden masters against merged main (#1985 + dockerfile class fix) * Regenerate golden masters (2nd merge-race) and fix ruff baseline line-shift Second conflict round: #1987 (jcl/m4) landed on main right after the first resolution, requiring another regenerate-and-rebless pass. Also fixes a real, separate issue caught while investigating why PR #1985/#1987 merged despite a failing ruff-audit CI check: #1972 (record_keeper.py's class_count fix, merged earlier and unrelated to this PR) shifted 7 pre-existing baselined findings by a couple of lines each -- tests/ruff_audit_baseline.json was never regenerated to absorb that shift, so every PR's ruff-audit CI run since has been reporting those 7 as spurious 'new' findings. Confirmed via audit_check.py that all 7 are pure line-shifts (same file/code/ message, just moved), not real regressions, and regenerated the baseline to absorb them. Verified against the exact CI-pinned ruff version (0.16.0, not whatever floats on PATH locally). --------- Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Gemini 3.1 Pro <noreply@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Same bug shape as dockerfile's own fix (#1976, merged):
jclandm4have noScopeParsingRegistryentry and no brace-delimited bodies at all, so they silently fell through toMode_B_Bracesin_function_slice-- which only produces a named function when a literal{happens to appear by coincidence within its search window.func_startmatches// <name> EXEC ...job-step lines -- JCL is fixed-column mainframe syntax with no brace concept anywhere.func_startmatchesdefine(...)/m4_define(...)/AC_DEFUN(...)etc. -- macro definitions are parenthesis-delimited with backtick/bracket quoting, not brace-delimited.Evidence
Direct measurement (
struct_func_start, the raw signal, vs.function_count, the named list actually reaching consumers), against the reallanguage-cruciblecorpus:Fix
Routes both to Mode A (
_slice_by_labels), the same "greedy to the nextfunc_startmatch" heuristic already proven for abap/cobol/fortran/assembly/dockerfile. Post-fix, both hit 100% recall in the local corpus (3/3, 39/39).makefilewas also flagged as a candidate in the same investigation but its corpus sample was inconclusive (only 1 real match total) -- deliberately left untouched, out of scope for this fix.Verification
pytestacross jcl/m4 extraction + strict tests, plustest_detector.py: 321 passed, 3 xfailed, 3 xpassed (pre-existing, unrelated).ruff_audit.py --ci/mypy_audit.py --ci: clean against baseline.crucible_check.py(full_precision+zero_dependency): real, expected drift in jcl/m4 files (new real functions captured, e.g.curl-sysconfig.m4'sCURL_DARWIN_SYSTEMCONFIGURATION) plus global spatial-layout ripple across the corpus (a real function-count change in a directory group shifts the whole graph's spatial layout, confirmed the same pattern as prior sweep sessions); both golden masters re-blessed; final re-run: PASS/PASS.Implemented via a Gemini/agy dispatch from a fully diagnosed issue (root-caused and filed in the same session as #1976), independently re-verified end to end (diff read, tests/audits/recall/crucible_check all re-run from scratch, not just trusting the subagent's own self-report) before this PR.
Fixes #1975.
🤖 Generated with Claude Code