Skip to content

Bound Mode A's per-function args search to the label's own statement - #1985

Merged
squid-protocol merged 1 commit into
mainfrom
fix/mode-a-args-bounding-1973
Aug 21, 2026
Merged

Bound Mode A's per-function args search to the label's own statement#1985
squid-protocol merged 1 commit into
mainfrom
fix/mode-a-args-bounding-1973

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

_calculate_block_metrics's generic per-function args_count derivation searched the entire greedy block when no args_search_text was passed. For Mode A (_slice_by_labels), that block legitimately spans from one func_start match to the next -- often many unrelated statements -- since Mode A has no brace-bounded body the way Mode B does.

For the 3 Mode A languages with no dedicated args_count_override (cobol, fortran, dockerfile), this let an unrelated, later occurrence get misattributed as the current label's own parameter count:

  • A dockerfile RUN could pick up an unrelated ARG line swept into its greedy span.
  • A cobol paragraph could pick up an unrelated CALL ... USING statement, or the file's own PROCEDURE DIVISION USING/RETURNING header -- confirmed on real corpus data: a paragraph literally named TIMESTAMP was showing 13 "arguments."

Fix

Adds _mode_a_args_window_end, which bounds the args search to the matched label's own statement span by following real line-continuation syntax (\ for dockerfile, & for fortran -- including Dockerfile heredocs and Fortran's own inline-comment/blank-line/cpp-directive interruptions, since WRF's real subroutines can have 200+ parameters spread across 90+ continuation lines) rather than a blind character-count window.

Scoped explicitly to primary_lang_id in (cobol, fortran, dockerfile) -- deliberately NOT args_count_override is None, which also fires for abap/agc_assembly/assembly whenever their own override legitimately resolves to None (e.g. an ABAP interface-implemented method with no DEFINITION-section entry). This distinction turned out to matter in practice: an earlier version of this fix gated on the override alone and silently applied an untuned bound to real ABAP methods, regressing their correct args counts (zcl_abapgit_http_client.clas.abap's check_http_200/send_receive: real 1, regressed to 0) -- caught by this PR's own crucible_check.py run before it shipped, then fixed to gate on language explicitly instead.

One known, narrow, separately-tracked residual

fortran's init_domain (module_initialize_real.F) drops from an accidental 1 to 0. Root cause: its own func_start match's match.start() lands on an unrelated earlier statement (INTEGER :: internal_time_loop, several blank lines before the real SUBROUTINE init_domain line) due to a separate, pre-existing bug in fortran's func_start regex -- filed as #1982. The old unbounded search happened to still reach the real signature further into an inflated span; a correctly bounded search anchored at the same (wrong) match start can't. Will self-resolve once #1982 lands.

Verification

  • pytest across dockerfile/cobol/fortran/abap extraction + strict tests, plus test_detector.py: 672 passed.
  • ruff_audit.py --ci / mypy_audit.py --ci: clean against baseline.
  • Real corpus before/after (galaxyscope --db-only against language-crucible/data/{dockerfile,cobol,fortran,abap}):
    • dockerfile: all functions now correctly show 0 args (was: several RUNs showing spurious 2).
    • cobol: 3 spurious nonzero-args paragraphs -> 0 (was: TIMESTAMP=13, ACCOUNT-OVERDRAFT-COUNT=1, LOCAL-STORAGE=1; all clearly bogus for paragraphs with no formal parameter concept).
    • fortran: 184/185 real functions unchanged (including WRF's lsm/lsm_mosaic, 247/291 real dummy arguments correctly preserved); 1 known residual traced to fortran func_start's optional TYPE-prefix can bridge across an unrelated statement to a distant SUBROUTINE/FUNCTION #1982.
    • abap: confirmed unaffected/correct (check_http_200=1, send_receive=1, set_header=4, matching the pre-existing golden master).
  • crucible_check.py (full_precision + zero_dependency): real, expected drift only in dockerfile/cobol/fortran files plus global-aggregate ripple; both golden masters re-blessed; final re-run: PASS/PASS.

This is #1973 in the same dockerfile tri-comparison manual-verification session as #1976 (merged) -- found while re-verifying the dockerfile func_start fix's own downstream effects, generalized once cobol/fortran turned out to share the exact same unbounded-search code path.

Fixes #1973.

🤖 Generated with Claude Code

_calculate_block_metrics's generic args-count derivation searched the
WHOLE greedy block when no args_search_text was passed -- for Mode A
(_slice_by_labels), that block can span many unrelated statements past
the matched label's own signature, since Mode A's body legitimately
runs to the next func_start match, not to a brace-bounded end.

For cobol/fortran/dockerfile (the only 3 Mode A languages with no
dedicated args_count_override), this let an unrelated, unbounded later
occurrence get misattributed as the current label's own parameter
count: a dockerfile RUN could pick up an unrelated ARG line swept into
its greedy span, and cobol paragraphs could pick up an unrelated
CALL...USING statement or the file's own PROCEDURE DIVISION
USING/RETURNING header -- confirmed on real corpus data (a cobol
paragraph literally named TIMESTAMP was showing 13 "arguments").

Adds _mode_a_args_window_end, which bounds the search to the label's
own statement span by following real line-continuation syntax
(backslash for dockerfile, ampersand for fortran, including Dockerfile
heredocs and Fortran's own inline-comment/blank-line/cpp-directive
interruptions -- WRF's real subroutines can have 200+ parameters
across 90+ continuation lines) rather than a blind character count.

Explicitly scoped to primary_lang_id in (cobol, fortran, dockerfile)
-- NOT "args_count_override is None", which also fires for
abap/agc_assembly/assembly whenever their own override legitimately
resolves to None (e.g. an ABAP interface-implemented method with no
DEFINITION-section entry). Verified this distinction matters: an
earlier gate on the override alone silently applied an untuned bound
to real ABAP methods and regressed their correct args counts -- caught
by this fix's own crucible_check verification before it shipped.

One known, narrow, separately-tracked residual: fortran's
init_domain (module_initialize_real.F) drops from an accidental 1 to
0, because its own func_start match.start() lands on an unrelated
earlier statement due to a separate, pre-existing func_start regex bug
(filed as #1982) -- the old unbounded search happened to still reach
the real signature further into an inflated span; a correctly bounded
search anchored at the same wrong start can't. Will self-resolve once
#1982 lands.

Fixes #1973.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@squid-protocol
squid-protocol merged commit 379449c into main Aug 21, 2026
25 of 27 checks passed
@squid-protocol
squid-protocol deleted the fix/mode-a-args-bounding-1973 branch August 21, 2026 01:41
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

Severity Count
🔴 Critical 0
🟠 High 0
🟡 Medium 0
🟢 Low 0
ℹ️ Info 3

ℹ️ Info Findings

[checkov] Ensure that a user for the container has been created

File: /docs/language_status/dockerfile.md:1
Rule: CKV_DOCKER_3
Ensure that a user for the container has been created

[checkov] Ensure that HEALTHCHECK instructions have been added to container images

File: /docs/language_status/dockerfile.md:1
Rule: CKV_DOCKER_2
Ensure that HEALTHCHECK instructions have been added to container images

[checkov] Ensure the base image uses a non latest version tag

File: /docs/language_status/dockerfile.md:157
Rule: CKV_DOCKER_7
Ensure the base image uses a non latest version tag

🐦‍⬛ Powered by Muninn · Skald Lab

squid-protocol added a commit that referenced this pull request Aug 21, 2026
…1987)

* Fix jcl and m4 func_start recall: route to Mode A, not brace search

Same bug shape as dockerfile (#1976): jcl and m4 have no
ScopeParsingRegistry entry and no brace-delimited bodies at all, so
they silently fell through to Mode_B_Braces in _function_slice --
which only produces a named function when a literal { happens to
appear by coincidence within its search window.

jcl's func_start matches "// <name> EXEC ..." job-step lines -- JCL is
fixed-column mainframe syntax with no brace concept anywhere. m4's
func_start matches define(...)/m4_define(...)/AC_DEFUN(...) etc. --
macro definitions are parenthesis-delimited with backtick/bracket
quoting, not brace-delimited.

Confirmed via direct measurement (struct_func_start, the raw signal,
vs. function_count, the named list actually reaching consumers):
  jcl: 3 raw matches, 0 reached the named list (0% recall)
  m4:  39 raw matches, 1 reached the named list (2.6% recall)

Routes both to Mode A (_slice_by_labels), the same greedy-to-next-
match heuristic already proven for abap/cobol/fortran/assembly/
dockerfile. Post-fix: both hit 100% recall in the local corpus (3/3,
39/39). makefile was also flagged as a candidate in the same
investigation but its corpus sample was inconclusive (1 match total)
-- deliberately left untouched, not part of this fix's scope.

Verified against the full ~80-repo crucible corpus (both golden
masters re-blessed; drift is real function-count/structural-mass
changes in jcl/m4 files plus expected global spatial-layout ripple
across the corpus, not corruption elsewhere).

Fixes #1975.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Gemini 3.1 Pro <noreply@google.com>

* Regenerate golden masters against merged main (#1985 + jcl/m4 fix)

---------

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Gemini 3.1 Pro <noreply@google.com>
squid-protocol added a commit that referenced this pull request Aug 21, 2026
…-shift

Second conflict round: #1987 (jcl/m4) landed on main right after the
first resolution, requiring another regenerate-and-rebless pass.

Also fixes a real, separate issue caught while investigating why
PR #1985/#1987 merged despite a failing ruff-audit CI check: #1972
(record_keeper.py's class_count fix, merged earlier and unrelated to
this PR) shifted 7 pre-existing baselined findings by a couple of
lines each -- tests/ruff_audit_baseline.json was never regenerated to
absorb that shift, so every PR's ruff-audit CI run since has been
reporting those 7 as spurious 'new' findings. Confirmed via
audit_check.py that all 7 are pure line-shifts (same file/code/
message, just moved), not real regressions, and regenerated the
baseline to absorb them. Verified against the exact CI-pinned ruff
version (0.16.0, not whatever floats on PATH locally).
squid-protocol added a commit that referenced this pull request Aug 21, 2026
* Add named class (build-stage) extraction for dockerfile

Dockerfile had no named class extraction at all: it wasn't in
_CLASS_START_NAMED_EXTRACTION_LANGS, so it fell back to the legacy
generic regex (class|struct|interface|trait|enum), which never
matches Dockerfile's FROM syntax at all -- correctly, since Dockerfile
has none of those keywords. Result: class_data stayed completely
empty for every Dockerfile scanned, even though struct_class_start
(the raw signal) correctly counted every real FROM line.

class_start's own regex previously captured only the literal keyword
FROM itself, not the build stage's real name -- extended it to an
alternation shape (matching the existing Fortran/Lua/ABAP convention
_resolve_class_start_match already documents): group 1 captures the
AS <alias> name when present, group 2 falls back to the bare base-image
reference for a stage with no alias (the file's final/default stage).
--platform=$VAR-style flags between FROM and the image reference are
skipped over. Added dockerfile to _CLASS_START_NAMED_EXTRACTION_LANGS,
and to the boundary-resolution skip condition alongside abap (same
#1907 rationale -- Dockerfile stages are never nested, and the brace
search would mistake ${VAR} template-substitution braces for a real
body opener).

Verified: class_data row counts now match struct_class_start exactly
(69/6/1/1 across the moby test corpus), with real stage names (base,
criu, xx, binary-dummy, ...) instead of the literal string "FROM"
repeated. ReDoS-probed the new regex directly (5000-char runs, deeply
nested continuations, huge flag/image tokens) -- all sub-millisecond.

One known, accepted side effect: THE LINEAGE EXTRACTOR (detector.py's
generic "any class_start match with 2+ groups treats group 2 as an
inheritance parent") doesn't know groups 1/2 here are alternation-
exclusive, not name-then-parent -- a bare FROM <image> (no alias)
sweeps the image reference into that file's parent_entity metadata.
This is a pre-existing, already-shipped pattern (fortran's own
class_start has the identical alternation shape and triggers the same
behavior for bare TYPE declarations today), not something this fix
introduces -- tracked as its own issue (#1983) for a future, more
alternation-aware fix to the shared extractor, not blocking here.

Fixes #1974.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Gemini 3.1 Pro <noreply@google.com>

* Regenerate golden masters against merged main (#1985 + dockerfile class fix)

* Regenerate golden masters (2nd merge-race) and fix ruff baseline line-shift

Second conflict round: #1987 (jcl/m4) landed on main right after the
first resolution, requiring another regenerate-and-rebless pass.

Also fixes a real, separate issue caught while investigating why
PR #1985/#1987 merged despite a failing ruff-audit CI check: #1972
(record_keeper.py's class_count fix, merged earlier and unrelated to
this PR) shifted 7 pre-existing baselined findings by a couple of
lines each -- tests/ruff_audit_baseline.json was never regenerated to
absorb that shift, so every PR's ruff-audit CI run since has been
reporting those 7 as spurious 'new' findings. Confirmed via
audit_check.py that all 7 are pure line-shifts (same file/code/
message, just moved), not real regressions, and regenerated the
baseline to absorb them. Verified against the exact CI-pinned ruff
version (0.16.0, not whatever floats on PATH locally).

---------

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Gemini 3.1 Pro <noreply@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mode A (_slice_by_labels) generic args-count search is unbounded, can attribute unrelated text to a function's parameter count

1 participant