feat(trace-repair): size the gated-stop draw on its power curve and abort on a substituted model

[drewstone]

What this branch does

Sizes the gated-stop confirmatory draw on its power curve, aborts on a substituted model, and pins the arms to the model the seat actually serves.

The seal chain

The model id is carried in both arm pins, so re-pinning necessarily moves the digest. That is the seal doing its job, not drift.

	digest	outcome
previous, pinned glm-5.2	ad82d3424f9949a53f12f2b0a709d0acb6b61f6e6472ba15727b1667c2ad1b4d	aborted at the servedModel gate, $0 spent, 0 rows graded
current, pinned glm-5.3	e13a4f8d9ff437dbf7d24898d64cba55c90c682c3902ad30bb528ca241939626	gate passes, run executes

The gate was not loosened. Measured directly on the seat:

pinned glm-5.2 -> served glm-5.3
pinned glm-5.3 -> served glm-5.3

The provider retired glm-5.2 and aliased it. The pin was stale, the seat was honest, so the pin moved to the truth and the comparison stayed absolute. The runner re-executes seal e13a4f8d and logs servedModel gate: pinned=glm-5.3 served=glm-5.3 passed=true before the first row is graded.

The draw, chosen on the power curve

The admitted set is the ceiling, not the draw. settlingDraw returns the smallest draw clearing the registered target at the settling effect of 0.10.

• n = 151 rows over 22 clusters, power 0.8009 (search sim, 15,000 trials) and 0.8013 (registered sim).
• Ceiling, all 216 admitted rows: power 0.9189.
• Registered curve at the chosen draw: 0.05 => 0.345, 0.10 => 0.801, 0.15 => 0.972, 0.20 => 0.996.

Near the floor the two estimates straddle the target, so a draw must clear 0.80 under both before the search accepts it. 149 rows clears the search sim (0.8055) but fails the registered sim (0.7897) and is rejected; 151 clears both.

Denominator chain

Population: terminalbench-trajectories, mini-swe-agent rows with reward 0, on tasks whose oracle certification is checked in.

stage	in	excluded	left
certified-deterministic-oracle	2601	0	2601
replayable-commands-and-final-returncode	2601	1528	1073
stratum-carries-a-repairable-failure	1073	3	1070
unknown-returncode-ratio-at-most-25pct	1070	126	944
recorded-commands-at-most-25	944	164	780
image-pinned-by-digest	780	0	780
one-row-per-recorded-trial	780	172	608
recorded-end-state-fails-its-own-suite	608	324	284
prefix-divergence-at-most-10pct	284	66	218
not-exposed-by-the-mechanism-pilot	218	0	218
task-carries-at-least-two-rows	218	2	216
admitted			216
drawn on the power curve			151

Cluster structure: 22 clusters, min 2 rows, median 8, max 9.

The known contaminant is excluded

qemu-startup__PnXK6EH::ord0 records reward 0 but its own held-out suite passes on the replayed end state. The recorded-end-state-fails-its-own-suite stage catches it, and it is confirmed absent from the sealed draw.

Measured end-state disagreement, over 303 screened rows:

task	recorded 0 but end state passes
qemu-startup	1/18 (5.6%)
every other task	0/285 (0.0%)

qemu-startup still contributes 8 rows to the draw. One label in eighteen disagreeing on a single task, against zero in 285 elsewhere, is a localized label defect and not a corpus-wide one — that task's remaining labels can carry a study, but qemu-startup is the one cluster whose result should be read with the exclusion in mind.

The measured result (run completed after this PR opened)

This PR is provenance for a completed experiment. The confirmatory run finished both arms: 302 rows graded, 151 per arm.

• Corrected primary contrast (best intermediate, both arms): +0.0596, 95% CI [-0.0061, +0.1210]. The interval includes zero; the registered verdict is no-effect-resolved-at-this-n. The study does not certify a gated-stop advantage at this draw.
• 15 gated-arm rows were degraded by provider 429 responses; the number above is the tail audit's corrected value.
• The registered estimand pipeline as first pushed crashed at report time on a boolean-vs-number defect. This PR now writes the verdict as 1 or 0, and the contrast above was recomputed from the persisted confirm-runs.json.

docs/trace-repair-gated-stop.md carries the same result section.

[tangletools]

✅ Auto-approved drewstone PR — 4f035221

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-14T13:06:17Z}

[tangletools]

⚠️ Value Audit did not run — no verdict

This is not an approval and not a clean bill of health. Neither interrogation lens returned a judgment, so this PR has no value-audit evidence for or against it.

Status	audit-incomplete (could not run)
Why	value: cli-bridge admission rejected (queue saturated); usefulness: cli-bridge admission rejected (queue saturated)
Lenses answered	0 of 2
What to do	re-run once the CLI bridge has capacity: pr-reviewerctl trigger <repo>#<pr> --force

💰 Value — error

value agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/kimi-for-coding/k2p7: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=1/48 — no model was started

🎯 Usefulness — error

usefulness agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 4
• Bridge error: opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=0/48 — no model was started

No concerns are listed because nothing examined the change — absence of findings here is absence of evidence, not a pass.

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260814T131135Z}

Value audit could not run (value: cli-bridge admission rejected (queue saturated); usefulness: cli-bridge admission rejected (queue saturated)). This approval was provisional on that audit running, so it is dismissed. Re-run the reviewer once the CLI bridge has capacity.

[tangletools]

✅ Auto-approved drewstone PR — 9ed5171f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-14T13:31:20Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Coverage	1 of 2 lenses (usefulness)
Concerns	2 (2 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	477.4s (2 bridge agents)
Total	477.4s

⚠️ Partial audit — the verdict covers only usefulness. value: cli-bridge admission rejected (queue saturated). Treat the missing lens as unexamined, not as clear.

💰 Value — error

value agent never ran: the CLI bridge refused admission (no model was started).

• Model: opencode/deepseek/deepseek-v4-pro
• Bridge attempts: 7
• Bridge error: opencode/kimi-for-coding/k2p7: opencode: opencode error; opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=1/48 — no model was started

🎯 Usefulness — sound

The PR turns the sealed-experiment library's registered-but-never-executed identity gate into its first real executor, sizes the confirmatory draw on the library's own power engine instead of appetite, and converts a silent model substitution into a $0 abort with a checked-in verdict artifact — ever

• Integration: Fully wired and reachable. The servedModel gate kind has existed in the library since PR #564 (src/experiment/ast.ts:661, evaluator at ast.ts:726-738, dispatched through registered.gate at src/experiment/define.ts:462) and was previously exercised only by tests (tests/experiment/preregistrations.ts, tests/experiment/budget-and-seal.test.ts); this PR's confirm mode (scripts/tb-gated-stop-ab.t
• Fit with existing patterns: Fits the codebase's grain rather than competing with it. The repo already separates 'library computes and refuses' (clusteredPower + assertDesignAdequate) from 'study decides policy' — settlingDraw is study-local policy on top of the library engine, consistent with that split. There is an established served-model module at src/integrity/served-model.ts (checkServedModel/normalizeModelId, alias-tol
• Real-world viability: Holds up beyond the happy path. The dual-clearance design (a draw must clear the 0.80 floor under both the hot 15k-trial search and the registered 3k-trial gate, scripts/tb-gated-stop-ab.ts:356-364) is precisely the guard against Monte Carlo noise near the floor — the checked-in scanned curve proves the failure mode it prevents (149 rows: hot 0.8055, registered 0.7897 — fails). The gate fails clos
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 3

🎯 Usefulness Audit

🟡 Model identity is gated once pre-spend but never re-checked per step [robustness] ``

The servedModel gate runs on a single probe before grading (scripts/tb-gated-stop-ab.ts:1232-1238). Every step already records its served id (line 871) and the report aggregates them (line 1376), but nothing compares those per-step ids against the pin, so a substitution that begins mid-run lands in the contrast and appears only as a telemetry line in confirm-report.json. A one-line post-run assertion over the recorded servedModels set against MODEL would close the loop; as-is it does not gate sh

🟡 confirm does not assert its regenerated seal matches the checked-in design [robustness] ``

confirm rebuilds the seal from local inputs (corpus, end-state screen, pilot exposures) at scripts/tb-gated-stop-ab.ts:1037-1049. If any local state drifted since design.json was checked in, confirm would execute under a digest nobody registered, detectable only by manually comparing the report's sealDigest to benchmarks/trace-repair/gated-stop-ab/design.json after the run. An assertion that sealed.digest equals the checked-in design.json's sealDigest (the same file already read at lines 1075-10

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260814T133936Z}

[tangletools]

❌ Needs Work — 9ed5171f

Review health 100/100 · Reviewer score 0/100 · Confidence 80/100 · 41 findings (1 critical, 3 high, 9 medium, 28 low)

	opencode GLM 5.2	opencode DeepSeek v4 Pro	opencode DeepSeek v4 Flash	aggregate
Readiness	0	13	20	0
Confidence	80	80	80	80
Correctness	0	13	20	0
Security	0	13	20	0
Testing	0	13	20	0
Architecture	0	13	20	0

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision.

Blocking

🟣 CRITICAL confirm contrast feeds boolean passed to a numeric-only estimand and interval — scripts/tb-gated-stop-ab.ts

armRows builds EvidenceRecord with passed: passedOf(run) where passedOf returns run.bestPassed or finalPassed — both boolean. The registered estimand is pairedContrast with value:'passed' (line 514), and computeEstimand for kind 'paired-mean-diff' throws ValidationError('... value field ... is not a number') when typeof value !== 'number' (src/experiment/ast.ts:324-328). computeInterval cluster-bootstrap has the same number check (ast.ts:411-414). So registered.estimate('pairedContrast', evidence) at line 1361 throws

🔴 HIGH Errored runs are silently counted as completed contrast outcomes and never retried — scripts/tb-gated-stop-ab.ts

runRow catches every error (lines 892-894) and returns a RowRun with error set; the confirm arm-B loop persists it as completed (held[key] = await runRow(...), line 1291) and the resume path skips it forever (if (held[key] !== undefined) continue, line 1286). The contrast builder (armRows, [lines 1341-1350](https://github.com/tangle-network/agent-eval/blob/9ed5171f06291229994c7de16ed9810ec5a3cbba/scripts/

🔴 HIGH confirm mode crashes at the contrast stage: boolean passed is rejected by estimate and interval — scripts/tb-gated-stop-ab.ts

armRows() builds evidence rows with passed: passedOf(run) — a boolean (line 1349). computeEstimand (src/experiment/ast.ts:325) and computeInterval (ast.ts:412) both require typeof value === 'number' and throw otherwise. Verified by executing the sealed spec with boolean rows: estimate throws "paired-mean-diff value field 'passed' is not a number on pair 'r1'"; interval throws the analogous error. Impact: after the full blind+gated spend (projected ~$13, 1-2 days wall clock; the live WORK run already holds 10 rows and is headed there), confirm never writes confirm-report.json and exits on an unhandled ValidationError. Fix: map pass verdicts to 1/0 when building

🔴 HIGH pairedContrast95 interval is computed over the pooled arm pass rate, not the paired difference — scripts/tb-gated-stop-ab.ts

evidence concatenates both arms' per-row records and interval() is called with value 'passed'; computeInterval buckets by taskName and bootstraps the mean of ALL values, mixing arms. Repro: two pairs with diffs +1/-1 (true effect exactly 0) yield interval [0.5, 0.5] — excludes zero, so the registered decision rule (interval-excludes-zero, sign positive -> 'gated-stop-survives', lines 554-560) false-positives under the null whenever the pooled pass rate is significantly above zero. The power gate (clusteredPower) simulates per-row DIFFERENCES, so the design's interval and its power model disagree. Fix: build one evidence row per pair with value = treatmentPasse

Other

🟠 MEDIUM design.json omits the model pin that the seal hashes and that this PR's own refusal hinged on — benchmarks/trace-repair/gated-stop-ab/design.json

design.json is the only human-readable registration record checked in, yet it carries no arm pins: the sealed spec pins model: MODEL per arm (scripts/tb-gated-stop-ab.ts:407,419, MODEL='glm-5.3' at :94), and the sealDigest e13a4f8d hashes that pin. This PR's own confirm-refusal.json proves the pin is decision-relevant — a confirm run aborted at $0 because the pinned id (glm-5.2) did not match the served id, and the seal was rotated (dadf7516 -> ad82d342 -> e13a4f8d) specifically to re-pin it. An auditor reading design.json alone cannot verify which model the current registration pins; they must trust the script source. Fix: include the arm pins (model, stepAllotment, temperature) in the designReport object written at scripts/tb-gated-stop-ab.ts:1086.

🟠 MEDIUM Done-signal rates (51.69%/31.14%/62.5%) are unartifacted and do not compose under the natural reading — docs/trace-repair-gated-stop.md

The sentence claims the done-signal 'fires on 51.69 % of failed runs and 31.14 % of successes, which is 62.5 % precision as a success predictor' on this corpus. Three problems: (1) scope — the gated-stop corpus is by construction 100 % reward-0 rows (design.json funnel population), so '31.14 % of successes' cannot be measured on it; no checked-in artifact and no file in the work dir contains 51.69 or 31.14 (repo-wide grep finds only this doc; charter.md:93 carries the pre-existing bare 62.5 %). (2) composition — P(success|fired)=0.625 with the stated conditionals requires successes to outnumber failures 2.77:1, implausible for a terminalbench mini-swe-agent dump; with the labels swapped (31.14 % on failures, 51.69 % on successes) a ~1:1 corpus yields exactly 62.5 %, suggesting a label tran

🟠 MEDIUM Headline-selection rule ('harsher contrast is the headline') is prose, not a registered rule — docs/trace-repair-gated-stop.md

The doc commits: 'When the two disagree, the harsher contrast is the headline.' The runner computes and reports both contrasts (scripts/tb-gated-stop-ab.ts:1354-1364, 'C-vs-B-best' and 'C-vs-B-final'), but the sealed decision table in buildSpec (:547-583) branches on a single interval 'pairedContrast95' with no registered rule selecting which contrast feeds the verdict or which is the headline. docs/experiment.md:7-8 ('The registered rule is the executed rule... never a closure or prose') makes this exactly the kind of rule that must be a typed node under the seal; as written, the headline choice is a post-hoc manual decision — the garden of forking paths the seal exists to close. Fix: register the selection rule (e.g., decision reads min(best, final) contrast) in the spec, or soften the d

🟠 MEDIUM C-vs-B-final contrast uses best-pass for the treatment arm — scripts/tb-gated-stop-ab.ts

treatment is computed once with (run) => run.bestPassed and reused for both 'C-vs-B-best' and 'C-vs-B-final' (lines 1355-1358), while the control side of the final contrast uses finalPassed (last step's gradePassed). It is currently equivalent because the gated arm stops on pass so best==final, but the coupling is implicit and fragile: any future change to runRow stop semantics would silently make the labeled 'final' contrast compare gated-best vs blind-final. Pass finalPassed for the treatment in the 'C-vs-B-final' branch to match its label.

🟠 MEDIUM Confirm executor never enforces the registered power-floor halt, matched-budget refusal, or the decision table — scripts/tb-gated-stop-ab.ts

powerGate and halt are computed at lines 1067-1071 for the report only; the confirm block (1225-1392) spends up to COST_CEILING_USD without consulting them, even though the registered halt (spec lines 537-541) is action 'refuse-spend' and the file header (32-33) promises a structure that cannot see the effect 'refuses the spend'. Likewise the registered matchedBudget onFail is 'refuse-contrast' (spec 542-546), but contrasts are written (1361-1364) regardless of budgets.matched (only logged at 1390), and registered.decide() i

🟠 MEDIUM Resume state confirm-runs.json is not bound to the seal digest — scripts/tb-gated-stop-ab.ts

held is loaded purely by arm::rowId key (1265-1272) with no digest recorded alongside; on resume (1286, 1314-1317) runs from a different draw or seal that share rowIds are reused silently. The report records the fresh sealDigest and previousSealDigest (1379-1380) but nothing ties held to either, so a resumed confirm after a corpus or oracle-registry change mixes runs from two registered designs into one contrast. Fix: persist the sealDigest in confirm-runs.json and refuse to resume when it does not match the freshly computed sealed.digest.

🟠 MEDIUM confirm never asserts its recomputed seal continues the checked-in design's chain — scripts/tb-gated-stop-ab.ts

confirm rebuilds the seal from live inputs (corpus, oracles, screen file, pilot-exposure files, MODEL) but never compares sealed.digest against the sealDigest recorded in benchmarks/trace-repair/gated-stop-ab/design.json, even though it reads that file for previousSealDigest (lines 1075-1084). If any input drifts (new pilot run adds exposed ids, corpus regenerated, oracle registry edit), confirm silently executes a different registration than the published design — the digest chain the repo's charter rests on is enforced only by the operator manually diffing hashes. Fix: abort in confirm mode when sealed.digest !== the checked-in design.json sealDigest unles

🟠 MEDIUM confirm resume state is not bound to the seal digest or pinned model — scripts/tb-gated-stop-ab.ts

held is loaded from confirm-runs.json keyed only by ${arm}::${rowId} with no sealDigest or MODEL recorded alongside. After a re-seal or model change — this PR itself changes MODEL from glm-5.2 to glm-5.3, which changes the seal digest — a resumed confirm run reuses runs produced under the prior registration inside the new seal, silently contaminating the contrast. The servedModel gate only inspects the fresh probe (line 1232), not held runs. Fix: persist and check the sealDigest (and MODEL) with the runs, and refuse or clear on mismatch.

🟠 MEDIUM servedModel identity gate checked once on a probe; per-step served models are never gated — scripts/tb-gated-stop-ab.ts

The gate runs only on the pre-run probe (lines 1232-1238). runRow records servedModel per step (line 871) but nothing re-evaluates registered.gate('servedModel', ...) over those records; the report merely lists unique servedModels post hoc (line 1376). The MODEL comment ([lines 89-92](https://github.com/tangle-network/agent-eval/blob/9ed5171f06291229994c7de16ed9810ec5a3cbba/scripts/tb-gated-stop-ab.ts#L89-L9

🟡 LOW Refusal artifact is a stale seal/pin that can be misread against the current design — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

confirm-refusal.json is checked in alongside design.json but describes a different, superseded seal. Its sealDigest (ad82d342...) and gate.evidence.pinned ('glm-5.2') belong to the pre-re-pin design; the current design.json has sealDigest e13a4f8d... and pins 'glm-5.3' (scripts/tb-gated-stop-ab.ts:94 MODEL='glm-5.3'). A reader pairing the two files can conclude the current design was refused. The only link is design.json's previousSealDigest=ad82d342; the refusal has no forward pointer to the superseding seal. Impact: documentation/audit ambiguity, no runtime effect (no code reads confirm-refusal.json). Fix: add a field (e.g. supersededBySealDigest) or a note in the refusal that the pin moved to glm-5.3 under seal e13a4f8d.

🟡 LOW Refusal references a ghost seal whose design content is not in the repo — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

sealDigest ad82d342 is neither the base seal (dadf7516) nor the head seal (e13a4f8d); the intermediate design.json carrying it was never committed, so the refused design's row set, power curve, and funnel are unrecoverable from git history. The material facts of the refusal event are self-contained (pinned glm-5.2, served glm-5.3, $0, 0 rows) and the chain link exists via design.json.previousSealDigest, so this is an auditability nit rather than a defect — but a reader can only infer, not verify, that the refused design differed from the current one solely in the model pin.

🟡 LOW spentUsd: 0 omits the token spend of the identity-gate probe — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

The confirm flow makes a real model call before writing this refusal: scripts/tb-gated-stop-ab.ts:1232 const probe = await callModel([{ role: 'user', content: 'reply with the single word ok' }]) runs before the gate check at line 1239 and the refusal write at line 1251. The seat was therefore billed prompt+completion tokens, yet the artifact records spentUsd: 0. The note scopes this to 'no row graded and no arm run', so it is accurate for study spend and not a silent data cor

🟡 LOW Chosen draw clears the power floor by a hair at the settling effect — benchmarks/trace-repair/gated-stop-ab/design.json

settlingN.rows=151 clears power 0.8 at effect 0.10 with margin 0.0009 (settle, 15000 trials) and 0.0013 (registered re-check, 3000 trials). Scan point rows=149 was rejected because its registered estimate (0.7897) dipped below target while its hotter estimate (0.8055) cleared, showing the two estimates straddle the floor near the chosen n. This is the documented intent of the double-clear rule (docs/trace-repair-gated-stop.md: 'A draw must clear the target under both before the search accepts it'), so it is not a defect; it is a design decision to buy only the power the design registered. Reviewer should be aware the operating margin is ~1 sigma of the 3000-trial Monte Carlo error, so a re-seal with a different seed or corpus could land just below 0.8 and trigger the powerFloor gate.

🟡 LOW Draw clears the registered power floor by 0.0013 at the settling effect — benchmarks/trace-repair/gated-stop-ab/design.json

At effect 0.1 the registered power is 0.80133 vs target 0.8 — a margin smaller than one Monte Carlo SE (~0.007 at 3000 trials). The artifact's own settlingN scan shows adjacent draws straddling the floor (149 rows: hot 0.8055 / registered 0.7897 fails; 150 rows: 0.7961 fails). The dual-clear rule (both 15000-trial and 3000-trial sims must pass) and the 'buy the registered power and nothing beyond' rationale make this a deliberate, documented choice, but the MDE=0.1 at exactly 0.8 power claim is fragile; a confirmatory null at effect ~0.1 will be partially attributable to sizing luck. No change required; consider a one-row safety margin or noting the thin margin in the docs.

🟡 LOW Live bench-cache design.json diverges from the checked-in copy in timestamps only — benchmarks/trace-repair/gated-stop-ab/design.json

The working-copy artifact at /home/drew/bench-cache/gated-stop-ab/design.json was regenerated at 13:26:10 (after the checked-in 13:20:36 copy was committed) and carries previousSealDigest=e13a4f8d... equal to its own sealDigest, because the spec is deterministic and re-sealing the identical rows/take/model yields the identical digest. The checked-in file's chain (previousSealDigest=ad82d342 -> e13a4f8d) is the correct one and matches confirm-refusal.json. No content difference exists (seal, rows, clusters, power all identical); purely a provenance artifact of the regenerator reading its own digest from the file it replaced. No action required.

🟡 LOW Measured determinism numbers lack a committed raw artifact — docs/trace-repair-continuation.md

The new text asserts '19 of 20 replies to one identical prompt were distinct on glm-5.3 and 8 of 20 on glm-4.7'. The numbers match the table in docs/trace-repair-gated-stop.md:57-58 and the 'z.ai seat' wording matches gated-stop.md:51, but no raw output, probe script, or JSON artifact backing a 20-reply temperature-0 run exists in the repo (grep for '20 replies'/'replies distinct' found only docs; audit478/determinism.mjs measures verdict determinism, not model sampler determinism). For a doc, the cross-doc consistency is sufficient; provenance would be stronger with a committed measurement artifact or the probe script referenced.

🟡 LOW Served-vs-requested id aliasing omitted from cited numbers — docs/trace-repair-continuation.md

The cell names glm-5.3 and glm-4.7 directly, but the source (docs/trace-repair-gated-stop.md lines 55-56) records that glm-5.2/5.1/5 are all answered by glm-5.3 and that glm-4.7 is answered by itself. The numbers themselves are correctly transcribed; only the id-alias caveat is dropped. Since this row already defers to trace-repair-gated-stop.md for the threat-to-validity detail, this is informational, not a factual defect.

🟡 LOW Table sentence exceeds STE descriptive-sentence length limit — docs/trace-repair-continuation.md

The rewritten cell: 'Removes the sampler as a source of variation the policy controls. It does not make a continuation repeat: measured against the z.ai seat, 19 of 20 replies to one identical prompt were distinct on glm-5.3 and 8 of 20 on glm-4.7.' The final sentence runs ~34 words, over the repo's 25-word descriptive-sentence STE limit declared in CLAUDE.md. Content is accurate; a split into two sentences would satisfy the style rule. Non-blocking.

🟡 LOW Table-cell sentence exceeds the repo's 25-word STE descriptive limit and has an ambiguous count — docs/trace-repair-continuation.md

The middle sentence ('It does not make a continuation repeat: measured against the z.ai seat, 19 of 20 replies to one identical prompt were distinct on glm-5.3 and 8 of 20 on glm-4.7.') is ~30 words, over the 25-word descriptive limit the repo's CLAUDE.md mandates for doc prose. Also '19 of 20 replies were distinct' is ambiguous: it can read as 19 distinct replies among 20 (one duplicate pair) or 19 of 20 differing from the first reply; the load-bearing claim (not deterministic at temperature 0) holds either way and the phrasing matches the sibling doc, but splitting the measurements into their own sentence and stating the accounting (e.g. '20 identical prompts produced 19 distinct replies') would remove both issues. No functional impact.

🟡 LOW 'Adjacent draws differ by less [than the SE]' is contradicted by the checked-in scan — docs/trace-repair-gated-stop.md

design.json settlingN.scanned shows measured adjacent-draw power gaps near the floor of +0.0108 (rows 148→149: 0.7947→0.8055) and −0.0094 (149→150), both larger than the 0.007 standard error cited. The claim holds only for the noise-free row-spacing, not the measured curve the doc's own artifact publishes. Reword to 'the true spacing between adjacent draws is smaller than the standard error' or drop the comparison.

🟡 LOW 'Harsher contrast is the headline' is a prose-only rule not enforced by code — docs/trace-repair-gated-stop.md

The doc says the control arm is graded at final and best state, 'both contrasts are reported', and 'when the two disagree, the harsher contrast is the headline.' The runner (tb-gated-stop-ab.ts confirm) does write both contrasts (C-vs-B-best and C-vs-B-final) and reports them, but no code selects a harsher headline or runs the registered decision table over either contrast, so the honesty rule exists only in prose. A later report could headline the favorable contrast without violating any code path. Impact: the doc describes a discipline the artifacts do not enforce. Fix: have the confirm report writer select and label the harsher contrast, or register which contrast the decision rule consumes.

🟡 LOW Detection-quality numbers have no committed source and imply an unreported base rate — docs/trace-repair-gated-stop.md

The doc claims the done-signal 'fires on 51.69 % of failed runs and 31.14 % of successes, which is 62.5 % precision as a success predictor.' The three numbers are jointly only self-consistent at a base success rate of ~73.5% (0.3114p / (0.3114p + 0.5169(1-p)) = 0.625 => p = 0.7345). The study's own admission population is exclusively reward-0 rows (funnel.population: 'mini-swe-agent rows with reward 0', input 2601), i.e. recorded failures, so 'on this corpus' is at best ambiguous and at worst overstates what the admitted set supports. No artifact in the PR carries the 51.69%/31.14% measurement (only charter.md:93 carries 62.5%). Impact: a reader cannot audit the claim or its population. Fix: cite the measurement artifact, or state the base success rate and population the rates were measure

🟡 LOW Incomplete sentence in temperature-0 paragraph — docs/trace-repair-gated-stop.md

'A paired design that assumes it must treat run-to-run variation as a threat to validity and report it.' is a sentence fragment (missing main clause / a 'that' after 'assumes'). Reader must guess the intended meaning (a paired design that assumes determinism must treat run-to-run variation as a threat and report it). Grammar nit only; no factual impact.

🟡 LOW Precision figure is inverted relative to the stated fire rates — docs/trace-repair-gated-stop.md

Line 9 states the done-signal 'fires on 51.69 % of failed runs and 31.14 % of successes, which is 62.5 % precision as a success predictor.' The only arithmetic joining 51.69 and 31.14 that yields ~62.5% is 51.69/(51.69+31.14)=62.4%, which is the fraction of fires on FAILED runs. The success-side fraction is 31.14/(51.69+31.14)=37.6%. Since the signal fires on a larger share of failures than successes, it is negatively associated with success, so '62.5% precision as a success predictor' is directionally wrong under the natural reading P(success|fire). Either the precision should be labeled a failure predictor, or the success-precision number is 37.6%, or a base ra

🟡 LOW Seat-behavior measurement rows lack a committed measurement artifact — docs/trace-repair-gated-stop.md

The 'Measured seat behaviour' table (lines 55-60) states glm-5.2/glm-5.1/glm-5 are served by glm-5.3, glm-4.7/glm-4.6 serve themselves, and rates of 6/20 and 4/24 for multi-action-block first turns. Only the glm-5.2 -> glm-5.3 substitution is evidenced in the PR (confirm-refusal.json: evidence.pinned 'glm-5.2', served 'glm-5.3'); the glm-5.1, glm-5, glm-4.7, glm-4.6 probe results and the two action-block rates have no committed artifact, so they are not auditable from the repository. Impact: unverifiable measurement claims in a doc that otherwise anchors every number to a committed artifact. Fix: commit the probe outputs (as for confirm-refusal.json) or mark

🟡 LOW Seat-behaviour table: 3 of 6 rows have no auditable artifact — docs/trace-repair-gated-stop.md

Rows 'glm-5.2/5.1/5 answered by glm-5.3' (3 ids), 'glm-4.7/4.6 answered by themselves' (2 ids), and 'first turn draws more than one action block, 6 of 20 replies, both models' have no supporting artifact in the repo or in /home/drew/bench-cache/gated-stop-ab (only glm-5.2→glm-5.3 is incidentally evidenced by confirm-refusal.json; 19/20 and 8/20 are corroborated by trace-repair-continuation.md:18; 4/24 exists only in the untracked mechanism-pilot.json). These facts justified the MODEL='glm-5.3' pin (script:94) and 'bound what the study can claim' — per the repo's own evidence doctrine they should be a checked-in probe artifact or carry a link to one.

🟡 LOW --take flag can inject NaN into the sealed selection — scripts/tb-gated-stop-ab.ts

take = Number(process.argv[takeArg + 1]) yields NaN when --take is the last argument or non-numeric. NaN flows into selections.confirmatory.take; hashJson canonicalization renders NaN as null, so the sealed take becomes null and runSelectionRule (chosen.length < rule.take, ast.ts:204) selects zero rows, later crashing clusteredPower on empty cluster sizes. Validate Number.isInteger(take) && take > 0 and reject otherwise.

🟡 LOW --take parsing is unguarded: NaN or 0 crash the design; oversized values silently override the power-sized draw — scripts/tb-gated-stop-ab.ts

take = takeArg > 0 ? Number(process.argv[takeArg + 1]) : settling.rows (1034-1035). --take with no value yields NaN; take=NaN or 0 seals a selection that returns [], then clusteredPower throws at line 1067 ('clusterSizes must be positive integers, got []'). A value above admittedDraw.length overrides the power-sized draw while the report still prints settling.rows/maxPower (1107-1108). Fix: when --take is present, validate a positive integer <= admittedDraw.length and reject otherwise.

🟡 LOW --take without a valid integer yields NaN take and an obscure downstream crash — scripts/tb-gated-stop-ab.ts

takeArg > 0 ? Number(process.argv[takeArg + 1]) : settling.rows — a trailing or non-numeric --take produces NaN; defineExperiment does not validate selection.take, runSelectionRule's chosen.length < NaN returns an empty draw, and the failure surfaces later as clusteredPower's 'clusterSizes must be positive integers' with no hint at the cause. Fix: parse with an integer guard and fail with a clear CLI error.

🟡 LOW C-vs-B-final grades the control arm at its final state, contradicting its registered gradedAt pin — scripts/tb-gated-stop-ab.ts

finalPassed reads run.steps.at(-1)?.gradePassed (1351) applied to the blind-continue control, while the arm's registered pin is gradedAt: 'best-intermediate-state' (line 410). The primary registered estimand pairedContrast uses bestPassed for both arms (1352-1356), so the registered decision is unaffected, but the reported C-vs-B-final contrast grades the control on a rule the seal does not register. Fix: keep it as an explicitly-unregistered audit contrast, or pin a final-state grading variant into the seal if it should drive a verdict.

🟡 LOW Cost ceiling is check-then-act across three workers and has no intra-run cap — scripts/tb-gated-stop-ab.ts

Arm-B workers read spent() then await runRow (1287-1291); because held is shared and the check precedes the await, up to SEAT_CONCURRENCY runs can be in flight past the ceiling. In arm C, extra = Math.floor(pool / perStep) (1325) can give one row a very large allotment with the ceiling checked only before the run, so a single run can overshoot by more than a row. Impact is bounded because the ceiling is deliberately set above projected spend (104-112), but the ceiling does not bound spend as advertised. Fix: reserve the projected cost before each run or cap extra.

🟡 LOW identity gate runs once on the probe, not per step — scripts/tb-gated-stop-ab.ts

The servedModel gate is evaluated only against the probe reply before spend. Per-step servedModel is recorded (StepRecord.servedModel, line 741) and surfaced in the report's servedModels set (line 1376), but a substitution after the probe would not abort the run — it would only appear post-hoc. Given this PR's entire purpose is to catch a substituted model, consider re-checking the gate per step or asserting the final servedModels set equals the pin before grading the contrast.

🟡 LOW previousSealDigest catch swallows malformed chain files to null — scripts/tb-gated-stop-ab.ts

The try/catch maps both a missing and an unparseable design.json to null, so a corrupt seal-chain file is reported as 'previousSeal=none' instead of failing loud — the repo's own no-silent-fallback doctrine. Fix: distinguish ENOENT (null) from parse errors (throw).

🟡 LOW roundRobinSizes sorts clusters with localeCompare; the registered selection sorts groups by code unit — scripts/tb-gated-stop-ab.ts

runSelectionRule orders groups with plain .sort() (src/experiment/ast.ts:201) while the settling model uses localeCompare (line 282). Identical for the ASCII kebab-case task names in this corpus, but for any non-ASCII name the two orders can differ and the settling curve would size a slightly different draw than the sealed selection executes. Fix: use the same comparator as the substrate.

🟡 LOW servedModel identity is checked on one probe; per-step substitutions are recorded but never refused — scripts/tb-gated-stop-ab.ts

The registered servedModel gate (onFail 'abort') is evaluated once against a single probe (1232-1237). Each step records servedModel (line 872-878) and the report aggregates servedModels (1376), but a model substitution that begins after the probe (seat fallback mid-run) is never aborted, weakening the header claim (39-42) that a substituted model aborts the run. Impact: a mixed-model contrast is possible and only detectable post-hoc in the report. Fix: also fail (or flag) the run when any step's servedModel disagrees with the pinned id, keeping the zero-spend probe abort in front.

🟡 LOW settling draw clears the power floor by a sub-SE margin — scripts/tb-gated-stop-ab.ts

The checked-in design.json shows the chosen n=151 has powerAtRegisteredSim=0.8013 vs the 0.8 target — a 0.0013 margin. The file's own comment states the registered sim's standard error near 0.8 is ~0.007 (lines 263-266), so the 'smallest draw that clears' is within Monte Carlo noise of the floor; a 3000-trial re-check cannot robustly distinguish n=149..151. Not a crash, and the registered powerFloor gate independently passes at effect 0.2, but the central 'smallest sufficient draw' claim is statistically underdetermined at these trial counts. Consider re-checking the chosen draw at higher trials (e.g. SETTLING_TRIALS) before registering it.

🟡 LOW settlingDraw returns a non-clearing draw when nothing clears, contradicting its contract; hot search runs in every mode — scripts/tb-gated-stop-ab.ts

If no scanned draw clears both checks, chosen stays at maxTake with power below target, but SettlingDraw.rows is documented as 'smallest draw whose power clears the target' — the design report would present a failing draw as the settling choice (the registered powerFloor halt still protects spend, so impact is reporting-only). Also the search (15000-trial sims, ~27 points) executes before mode dispatch, so screen/pilot invocations pay minutes of simulation for settling numbers that screen mode then writes into a provisional design.json. Fix: flag the no-clear case explicitly and compute settling only for modes that consume it.

---

_{tangletools · 2026-08-14T13:58:34Z · trace}

[tangletools]

❌ 4 Blocking Findings — 9ed5171f

Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-14T13:58:34Z · immutable trace}

[tangletools]

✅ Auto-approved drewstone PR — c108ed32

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-14T14:03:26Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Coverage	2 of 2 lenses (value, usefulness)
Concerns	4 (4 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	151.5s (2 bridge agents)
Total	151.5s

💰 Value — sound-with-nits

Sizes the confirmatory draw at the smallest n that clears the 0.8 power target (151 rows, dual-checked at search and registered sim), registers a servedModel identity gate that aborted the first confirm attempt at $0 on a substituted model, and writes the confirm executor composing the library's sea

• What it does: Three deltas in scripts/tb-gated-stop-ab.ts + its checked-in artifacts. (1) settlingDraw (script:335-392) walks the draw upward and returns the smallest row count whose simulated power clears 0.8 at effect 0.10, evaluated at 15,000 trials AND re-checked at the registered gate's 3,000 trials before acceptance (script:356-364) — the design.json draw shrinks 216 -> 151 rows over 22 clusters, with t
• Goals it achieves: Buy exactly the power the design registered and nothing more (216 admitted rows were overkill for a 0.8 target at 0.10 effect — ~30% fewer graded rows at lower cost for the same registered guarantee); refuse to spend on a seat serving a model nobody registered (the provider retired glm-5.2 and aliased it to glm-5.3 — a silent substitution would have made the sealed arms' model pin a lie); and comp
• Assessment: Good on its merits, and squarely in the codebase's grain. The change reuses the substrate rather than reinventing: the identity gate is a library primitive (src/experiment/ast.ts:661, evaluateIdentityGate via registered.gate in src/experiment/define.ts:462-466), power comes from clusteredPower (src/experiment/power.ts:106) for both the search and the registered gate, and the halt's `report: 'settl
• Better / existing approach: Searched for an existing draw-size/sample-size search (git grep settling|sampleSize|sample-size across src/, scripts/, docs/ — only the halt report field 'settling-n' in ast.ts:774 and this script), for existing identity/model-pin gates (the ast.ts gate kind predates this PR and is correctly consumed, not duplicated), and for another paired-arm executor (none; earlier tb-* scripts are single-arm
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound-with-nits

The PR delivers the previously-deferred confirmatory executor for the gated-stop A/B and sizes its draw via a power-curve search, both built on the sealed-experiment substrate exactly as the repo's doctrine requires; it was exercised for real (the checked-in refusal artifact aborts at zero spend on

• Integration: Reachable and already used. The new confirm mode (scripts/tb-gated-stop-ab.ts:1225-1392) is the confirmatory executor the prior commit explicitly deferred (removed header comment 'The confirmatory executor... is not written'); it runs both arms, enforces the cost ceiling, persists resumable state (confirm-runs.json, :1265-1273), and writes confirm-report.json. The servedModel identity gate it
• Fit with existing patterns: Fits the codebase's grain precisely. The refusal is evaluated through registered.gate('servedModel', ...) (:1233) rather than the free-function assertServedModel (src/integrity/served-model.ts) — correct under the sealed-experiment doctrine (docs/experiment.md: registered rule = executed rule; refusals live inside artifacts). The roundRobinSizes projection (:278-297) reproduces the substrate
• Real-world viability: Mostly robust: resumable per-row persistence, cost ceiling checked before each row in both arms (:1287,1319), seat concurrency capped at 3 with a waiter queue (:692-701), per-call abort deadline, per-row error capture into the record, and the pre-spend probe gates identity before any grading. Two edge gaps: (1) the servedModel gate fires once on a pre-spend probe (:1232) and is never re-evaluated
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

💰 Value Audit

🟡 roundRobinSizes re-projects the library's round-robin selection rule [duplication] ``

script:278-297 computes cluster sizes for hypothetical takes by re-implementing the pass-based round-robin that runSelectionRule implements at src/experiment/ast.ts:180-207 (lexical group order, one row per pass). Acceptable trade — re-sealing a spec per candidate draw to call select() would burn ~26 seals for the scan, and the chosen draw IS verified to reproduce from the sealed rule (script:1044-1049), so the search is only a sizing heuristic. But the two implementations can drift silently (e.

🟡 Registered decision table is never executed [against-grain] ``

The spec seals a decision table (script:547-583: gated-stop-survives / dies / no-effect / contrast-refused branches) and every other rule runs through registered.* (admit, select, gate, halt, estimate, interval, matchedBudgets) — but registered.decide (src/experiment/define.ts:372-388's RegisteredExperiment surface) is never called anywhere in the script (grep: no invocation). The confirm report carries contrasts and budget verdicts from which the verdict is readable by hand, so nothing is los

🎯 Usefulness Audit

🟡 servedModel gate is probe-once; mid-run substitution is recorded, not refused [robustness] ``

The gate runs on a single pre-spend probe (scripts/tb-gated-stop-ab.ts:1232) and the run never re-evaluates it, though every StepRecord already carries servedModel (:871) and the final report aggregates servedModels (:1376-1383). The provider aliasing incident this PR responds to (glm-5.2 retired mid-stream) could recur mid-run and spend up to the $40 ceiling before landing in a report nobody gated. Suggested: re-run registered.gate('servedModel', ...) over the collected per-step served ids befo

🟡 Two divergent same-model comparisons now coexist [problem-fit] ``

The substrate's registered identity gate compares by bare basename equality (src/experiment/ast.ts:731-732) while the established integrity module normalizes prefixes, @snapshot/:tier suffixes, and date suffixes (src/integrity/served-model.ts:85-99). A pinned-id/served-id pair can read as alias under one and abort under the other. Strictness inside the seal is defensible (fails closed), but a human may want the registered gate's evidence to carry the checkServedModel verdict alongside matched so

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260814T140615Z}

[tangletools]

❌ Needs Work — c108ed32

Review health 100/100 · Reviewer score 0/100 · Confidence 80/100 · 38 findings (1 critical, 2 high, 8 medium, 27 low)

	opencode GLM 5.2	opencode DeepSeek v4 Pro	opencode DeepSeek v4 Flash	aggregate
Readiness	0	38	17	0
Confidence	80	80	80	80
Correctness	0	38	17	0
Security	0	38	17	0
Testing	0	38	17	0
Architecture	0	38	17	0

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision.

Blocking

🟣 CRITICAL confirm mode crashes computing the contrast: boolean passed rejected by estimand and interval — scripts/tb-gated-stop-ab.ts

armRows maps runs to { rowId, taskName, arm, passed: passedOf(run) } where passedOf returns a boolean (run.bestPassed / steps.at(-1)?.gradePassed === true). The registered estimand pairedContrast (value: 'passed') goes through computeEstimand (src/experiment/ast.ts:325-329) which throws paired-mean-diff value field 'passed' is not a number on any non-number, and pairedContrast95 goes through computeInterval (ast.ts:412-415) which throws the same class of error. Verified by executing both against the exact evidence shape: both throw. The throw lands after both arms have fully run (line 1361-1363), so the run spends hours and up to COST_CEILING_USD=40,

🔴 HIGH Registered refuse-spend halt is never enforced before spending in confirm — scripts/tb-gated-stop-ab.ts

The spec registers halt any-gate-failed [powerFloor] -> refuse-spend, and main() computes powerGate/halt in the shared design flow (lines 1067-1071) and prints them — but the confirm block (lines 1225-1392) checks only the servedModel identity gate before opening spend. If the power gate fails (operator passes --take 4, or corpus drift shrinks admitted clusters), design.json records halt.fired=true action=refuse-spend and confirm spends anyway. This is exactly the failure class the file's header claims is prevented ('the p

🔴 HIGH confirm always crashes on the contrast: passed is boolean but registered estimators require numbers — scripts/tb-gated-stop-ab.ts

armRows (lines 1341-1350) maps each run to passed: passedOf(run) where passedOf returns a boolean (run.bestPassed / finalPassed). registered.estimate('pairedContrast', ...) and registered.interval('pairedContrast95', ...) then read field 'passed': computeEstimand throws 'value field passed is not a number on pair' (src/experiment/ast.ts:325-328) and computeInterval throws 'value field passed is not a number in cluster' (ast.ts:412-415). I reproduced both throws with the repo's own estimators. The confirm branch runs every row first (up to COST_CEILING_USD=40 of spend), persists confirm-runs.json, then exits 1 at report time; a resume re-runs the same crash

Other

🟠 MEDIUM Done-signal detection statistics are unreproducible and internally inconsistent — docs/trace-repair-gated-stop.md

The doc states: 'on this corpus the recorded done-signal fires on 51.69 % of failed runs and 31.14 % of successes, which is 62.5 % precision as a success predictor.' Verified against the trajectory dump (tbench-20260808 parquet, mini-swe-agent rows with steps): under the repo's own done-signal definition (endedOnSubmitSentinel, scripts/tb-corpus-rows.ts / src/trajectory-replay/steps.ts:249), the signal fires on 78.19% of the 4961 failed runs and 96.11% of the 1517 successes; clean-exit (final rc=0) gives 70.63%/87.28%. Both directions are the opposite of the doc's (signal fires MORE on successes), no definition yields 51.69/31.14, and the triple is arithmetically impossible at the actual class balance: with those fire rates, precision is 15.1% (or 32.9% transposed); 62.5% precision require

🟠 MEDIUM A truncated arm is scored as a full negative difference, not 'zero difference' — scripts/tb-gated-stop-ab.ts

armRows filters out runs that are absent for an arm, so a pair with a control run but no treatment run is fed to computeEstimand's paired-mean-diff with only the control side. The estimand (ast.ts:341) does sum += (slot.treatment ?? 0) - (slot.control ?? 0), so a missing treatment with a passing control (value 1) contributes -1 — a full negative, not zero. The COST_CEILING comment (lines 107-110) claims such pairs score as 'a zero difference ... dilutes the contrast toward null', which is wrong: truncation actively biases the contrast against the treatment, risking a manufactured 'gated-stop-dies'. Mitigated in practice by the ceiling being above projected s

🟠 MEDIUM Mid-run model substitution is recorded but never gated — scripts/tb-gated-stop-ab.ts

The identity gate probes the seat once before spend (line 1232). Every step then records servedModel, and the report lists the set of servedModels (line 1376) — but no gate or refusal is evaluated over the per-step served ids. A seat that serves glm-5.3 for the probe and a substitute mid-run produces a fully graded contrast where the contamination is only a data field in the artifact; nothing refuses or even flags it against the registered onFail: 'abort'. Also the probe's own tokens are not counted, so the refusal artifact's `spentUsd:

🟠 MEDIUM Registered decision rule is never executed; confirm report carries no verdict — scripts/tb-gated-stop-ab.ts

The sealed spec registers a full decision table (gated-stop-survives / gated-stop-dies / no-effect-resolved-at-this-n / contrast-refused-unmatched-budget) and RegisteredExperiment.decide exists for it, but no code in the file ever calls decide(). The confirm report ends at contrasts + budgets; the registered verdict is left as prose for a human to derive. Given the module covenant ('the registered rule IS the executed rule'), the decision table is currently dead registration. Fix: after computing contrasts and budgets, build the DerivedQuantities evidence and call registered.decide(), writing the outcome into confirm-report.json.

🟠 MEDIUM confirm mode never enforces the registered power-floor 'refuse-spend' halt — scripts/tb-gated-stop-ab.ts

main computes powerGate (line 1067) and halt=registered.halt([powerGate]) (line 1071) unconditionally, but the confirm executor (line 1225+) never reads either. The header claims (lines 31-33) the power floor 'refuses the spend instead of producing an interval nobody can read', and the

🟠 MEDIUM confirm never enforces the registered power-floor halt (refuse-spend) — scripts/tb-gated-stop-ab.ts

powerGate and halt are computed in the shared design section (lines 1067-1071) and only printed/written in design mode (lines 1099, 1110). The confirm branch (lines 1225-1392) checks only the servedModel identity gate before spending; halt.fired is never consulted. The seal registers halt action 'refuse-spend' for a failing powerFloor (spec [lines 537-541](https://github.com/tangle-network/agent-eva

🟠 MEDIUM registered decision rule is never executed in confirm — scripts/tb-gated-stop-ab.ts

The confirm report computes estimate + interval for two exploratory labels but never calls registered.decide(...). The sealed decision table (spec lines 547-583) that yields the registered verdicts ('gated-stop-survives' / 'gated-stop-dies' / 'no-effect-resolved-at-this-n' / 'contrast-refused-unmatched-budget') is dead code in the executor. The report ships raw intervals and leaves the verdict to a reader, so the run does not produce the registered decision and the matched-budget obligation branch ('contrast-refused-unmatched-budget') can never fire in the artifact. If the intervals are only a staging output, surface the decision outcome in the report instead

🟠 MEDIUM servedModel identity gate checks only the probe, not each graded step — scripts/tb-gated-stop-ab.ts

The gate is evaluated once on a single probe call (lines 1232-1238); the header (lines 39-42) claims a substituted model 'aborts the run'. But runRow records servedModel per step (line 871) and the report aggregates servedModels (line 1376) without re-running the gate. A seat

🟡 LOW Refusal artifact references a superseded design seal — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

confirm-refusal.json seals ad82d342, but the committed design.json in the same directory now seals e13a4f8d (glm-5.3 pin). The refusal is the record of the PRIOR design (glm-5.2 pin) aborting; the current design has never been confirmed. The chain is traceable (design.json.previousSealDigest === ad82d342, verified), but an automated consumer or casual reader could read 'refused' against the current design and conclude the experiment will never run. Suggest adding a superseded/status field or a seal-derived filename so a refusal of a prior design is distinguishable at a glance.

🟡 LOW Refusal records spentUsd: 0 although the identity-gate probe spent a sub-cent model call — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

confirm evaluates servedModel by issuing one real model call ('reply with the single word ok', scripts/tb-gated-stop-ab.ts:1232) before writing the refusal, and the writer hardcodes spentUsd: 0 (scripts/tb-gated-stop-ab.ts:1245); the note also says 'no spend'. The probe consumes roughly 10 input and 1 output tokens at the pinned pricing (~1e-5 USD), so the value is materially zero and the field's evident intent is arm spend, not gate spend. Under this repo's no-silent-zeros doctrine the cleaner fix is to report the probe cost (ModelReply would need to carry costUsd) or rename/annotate the field as arm spend so the zero is unambiguous.

🟡 LOW confirm-refusal.json references a superseded seal and no longer matches the current model pin — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

The artifact's sealDigest ad82d342... is the seal from commit 4f03522 under the glm-5.2 pin; the current design.json pins MODEL=glm-5.3 (seal e13a4f8d...). The refusal documents the abort that motivated the re-pin, and the design.json previousSealDigest links it, but the refusal file itself carries no field tying it to the model-pin change. A reader opening only confirm-refusal.json may attribute the refusal to the current glm-5.3 design rather than the superseded glm-5.2 pin. The note text is accurate but should name the pin change or point at the design seal it precedes. Documented in docs/trace-repair-gated-stop.md 'Measured seat behaviour', so informational.

🟡 LOW spentUsd: 0 understates the identity probe spend — benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json

The script hardcodes spentUsd: 0 in the refusal object (tb-gated-stop-ab.ts:1244), but the servedModel gate runs one callModel probe before grading (tb-gated-stop-ab.ts:1232), which consumes a nonzero (tiny) token count against the paid seat. The note's claim 'No row was graded and no arm was run' is accurate; only the absolute-zero spend figure is slightly imprecise. Negligible in magnitude but the field name overpromises.

🟡 LOW Current design.json does not expose the pinned model id its servedModel gate enforces — benchmarks/trace-repair/gated-stop-ab/design.json

The servedModel identity gate compares a pinned id against the served id (refusal evidence shows the shape: pinned/served/matched), and the v2->v3 re-seal exists solely to change that pin from glm-5.2 to glm-5.3. Yet v3's design.json records no model field: the only readable pin in the shot's files is the superseded glm-5.2 in confirm-refusal.json, and the current pin is legible only from the script const MODEL and the docs table. Integrity is intact (the pin is inside arms.pins and hashed into sealDigest e13a4f89), but an auditor reading the sealed artifact alone cannot tell which model the 151 rows will run on, or that v2 and v3 differ by the pin rather than the rows. Fix: include the arms' pins (or minimally the pinned model id) in the designReport the script writes.

🟡 LOW Funnel surviving (216) vs rowCount (151) gap not surfaced as a funnel stage — benchmarks/trace-repair/gated-stop-ab/design.json

The funnel stages end at surviving=216, yet rowCount=151 and power.totalRows=151. The 216->151 reduction is the settling draw (settlingN block), not an admission stage, so a reader tracing only the funnel cannot account for the 65 dropped rows; the settlingN block is the only pointer. Verified the draw is a legitimate registered round-robin subset, so this is a documentation consistency nit: consider a 'settling-draw' note on the funnel or a field naming rowCount as 'drawnRows' next to admittedRows.

🟡 LOW Settled draw sits knife-edge on the power floor with a noisy scan — benchmarks/trace-repair/gated-stop-ab/design.json

The chosen draw (rows=151) clears the 0.8 target at the registered sim with power 0.8013 — within the ~0.007 MC standard error of the target — and the scan is non-monotonic near the floor (rows 149 hot=0.8055 but registered=0.7897; rows 150 hot dips to 0.7961; rows 151 hot=0.8009). Re-execution reproduces all values exactly, and the design honestly records it, but a draw whose registered power differs from the target by less than one standard error should be acknowledged as sensitive to seed/trial count; at rows 154 the registered power is 0.814. Documented behavior, informational.

🟡 LOW powerFloor gate passes on grid max, not the sizing effect where power is at the floor — benchmarks/trace-repair/gated-stop-ab/design.json

powerGate.passed:true is computed by evaluatePowerFloorGate as maxPower over the effect grid, which is 0.9957 at effect 0.2. But the draw is sized at SETTLING_EFFECT=0.1, where the reported power is 0.8013 (hot 0.8009) — only 0.0013 above the 0.8 target and within the ~0.007 Monte Carlo SE at 3000 trials. A reader can misread 'powerFloor passed' as certifying the design is well-powered for the effect it actually targets, when at effect 0.1 the power is statistically indistinguishable from the floor. This is the documented 'smallest draw that clears' intent, so it is not a defect, but the artifact should make the sizing-effect margin explicit rather than resting the passed flag on the grid maximum.

🟡 LOW Sentence exceeds repo STE word limit — docs/trace-repair-continuation.md

'It does not make a continuation repeat: measured against the z.ai seat, 19 of 20 replies to one identical prompt were distinct on glm-5.3 and 8 of 20 on glm-4.7.' is ~31 words; the repo's CLAUDE.md caps descriptive sentences at 25 words. Split the measurement sentence off from the 'threat to validity' sentence, or drop the model-id repetition since the sibling doc carries the full table. Content is accurate and cross-backed; purely a style nit.

🟡 LOW Table cell mixes 'why it is fixed' with a validity note — docs/trace-repair-continuation.md

The third column is headed 'why it is fixed', but the temperature cell now carries two distinct things: the fixation rationale ('Removes the sampler as a source of variation the policy controls') plus a threat-to-validity observation that is already the subject of the linked gated-stop doc. This is informative and correctly cross-linked, not misleading, so it is a nit at most — no change required to merge.

🟡 LOW Table-cell sentence exceeds repo STE 25-word descriptive limit — docs/trace-repair-continuation.md

The new cell's second sentence ('It does not make a continuation repeat: measured against the z.ai seat, 19 of 20 replies... on glm-4.7.') runs ~33 words against the repo's STE limit of 25 for a descriptive sentence (CLAUDE.md 'Keep STE sentence limits'). Impact: style-only, no correctness issue; the density is arguably justified by the measurement payload. Fix: split into two sentences, e.g. 'It does not make a continuation repeat. Measured against the z.ai seat, 19 of 20 replies to one identical prompt were distinct on glm-5.3; 8 of 20 on glm-4.7.'

🟡 LOW 'Harsher contrast is the headline' is not represented in the report artifact — docs/trace-repair-gated-stop.md

Doc states 'When the two disagree, the harsher contrast is the headline.' The runner's confirm path (scripts/tb-gated-stop-ab.ts:1354-1388) computes both contrasts ('C-vs-B-best' and 'C-vs-B-final') and writes them verbatim into confirm-report.json, but emits no headline/verdict field and no logic selecting the harsher; the registered decision table also references a single 'pairedContrast95' name that does not map to two stored contrasts. A future reader could mistake the headline convention for enforced output. Fix: either note that headline selection is a manual reading convention, or have the report emit a primary-contrast field the runner actually computes.

🟡 LOW 'The harsher contrast is the headline' is an unregistered, unenforced convention — docs/trace-repair-gated-stop.md

The doc promises that when C-vs-B-best and C-vs-B-final disagree, 'the harsher contrast is the headline.' Both contrasts are computed and written to confirm-report.json (scripts/tb-gated-stop-ab.ts:1354-1364), but nothing registers or executes a headline rule: the decision table in the spec is never evaluated by the runner (no executeDecisionRule call in confirm mode), so verdict assembly is manual. Per docs/experiment.md's premise (registered rule = executed rule), the doc should either register the harsher-contrast rule in the sealed spec or explicitly mark this sentence as operator reading guidance rather than a rule of the experiment.

🟡 LOW 'ends the run at zero spend' and the refusal's spentUsd:0 ignore the identity probe's token cost — docs/trace-repair-gated-stop.md

The doc says the servedModel gate 'ends the run at zero spend' and the refusal 'records the dollars spent', and the checked-in benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json records spentUsd: 0. But confirm runs one model probe before the gate (scripts/tb-gated-stop-ab.ts:1232, callModel), which consumes real tokens and therefore nonzero dollars. The magnitude is sub-cent, but the artifact claims to record the dollars spent and this repo's doctrine forbids silent zeros. Fix: record the probe's prompt+completion tokens in the refusal (or state the zero means zero row/arm spend).

🟡 LOW Doc claims the decision table returns 'contrast-refused-unmatched-budget' and the contrast 'is not read', but the runner never executes the decision table — docs/trace-repair-gated-stop.md

The confirm path in scripts/tb-gated-stop-ab.ts (lines 1340-1389) computes both contrasts and budgets and writes them into confirm-report.json unconditionally; it never calls registered.decide(...), which is the only executor of the registered decision table (src/experiment/define.ts:382,426). The 'contrast-refused-unmatched-budget' verdict and the 'contrast is not read' behavior are a registered rule awaiting a downstream consumer, not behavior this PR's runner enforces — confirm-report.json always carries both contrasts even when budgets.matched is false. Doc overstates the shipped behavior; a reader relying on the runner to suppress an unmatched contr

🟡 LOW Garbled sentence in 'Measured seat behaviour' section — docs/trace-repair-gated-stop.md

'A paired design that assumes it must treat run-to-run variation as a threat to validity and report it.' The pronoun 'it' is ambiguous and the clause reads as the design assuming it must treat variation, obscuring the intended meaning (a design that assumes temperature-0 determinism must treat run-to-run variation as a threat). Reword for clarity, e.g. 'A paired design that assumes a repeated continuation must treat run-to-run variation as a threat to validity and report it.'

🟡 LOW Resume guarantee overstated for the concurrent control arm — docs/trace-repair-gated-stop.md

'Every finished row is written to confirm-runs.json before the next row starts' and 'An interrupted run costs the row in flight and nothing before it' hold for the serial treatment arm, but the control arm runs SEAT_CONCURRENCY=3 rows concurrently (scripts/tb-gated-stop-ab.ts:1278-1302): persist() fires after each row completes, so an interruption can lose up to 3 in-flight control rows, not one. Reword to 'at most the rows in flight (up to three in the control arm, one in the treatment arm)'.

🟡 LOW --take argument unvalidated; NaN flows into the seal — scripts/tb-gated-stop-ab.ts

const take = takeArg > 0 ? Number(process.argv[takeArg + 1]) : settling.rows — a missing or non-numeric value yields NaN, which is sealed into the spec without validation and only surfaces later as an opaque clusteredPower ValidationError (empty clusterSizes) at line 1057. Fail-loud, so low, but an explicit argument check (Number.isInteger && >= 1) at parse time would name the operator's mistake.

🟡 LOW --take without a numeric value yields NaN and a crash rather than a refusal — scripts/tb-gated-stop-ab.ts

const take = takeArg > 0 ? Number(process.argv[takeArg + 1]) : settling.rows — --take as the last argument (or a non-numeric value) makes Number(undefined)=NaN, which flows into buildSpec's selection take: NaN, producing an empty draw and a later clusteredPower ValidationError. Fail-loud doctrine prefers an explicit 'invalid --take' error. Low because the script is run manually.

🟡 LOW Corrupt confirm-runs.json silently restarts spend from zero — scripts/tb-gated-stop-ab.ts

The state loader wraps readFileSync+JSON.parse in catch { return {} }, conflating 'no prior state' with 'state file corrupt'. persist() uses plain writeFileSync (not atomic temp+rename), so a crash mid-write leaves truncated JSON; the next run then re-runs every row and re-spends up to the full ceiling, and the reported costUsd (recomputed from held runs) silently excludes the wasted spend. This is the try/catch { return null } pattern the repo's no-fallbacks doctrine forbids, on the one file that gates real money. Fix: write via temp file + renameSync, and on a parse failure of an existing file, fail loud instead of returning {}.

🟡 LOW Reallocation pool is sequential, so later rows get systematically more extra steps — scripts/tb-gated-stop-ab.ts

The pool starts at 0 and only grows as earlier rows clear early, so rows late in the round-robin order receive extra steps that earlier rows never could (lines 1310-1338). Row order is deterministic (lex-asc round-robin) so the asymmetry is reproducible, but if row difficulty correlates with taskName ordering the treatment effect is confounded with position. Consider randomizing/registering the pool-drain order or noting the order-dependence in the sealed design.

🟡 LOW Registered decision rule and matched-budget 'refuse-contrast' are never executed — scripts/tb-gated-stop-ab.ts

confirm computes contrasts (1354-1364) and budgets via registered.matchedBudgets (1371), but never calls registered.decide(...) to produce the sealed decision table's verdict, and never invokes the matched-budget refusal (onFail: 'refuse-contrast'). budgets.matched is only logged (1390). The seal registers a decision rule and an obligation that the executed path ignores, so 'registered rule = executed rule' does not hold end-to-end; the verdict is left to a human reading confirm-report.json. Fix: call registered.decide with the computed estimand/interval/budget results and write the verdict into the report.

🟡 LOW identity gate checks only the start probe; per-step servedModel is recorded but never gated — scripts/tb-gated-stop-ab.ts

The servedModel identity gate runs once on a single probe before grading. The docstring ('the id the seat reports on the reply and aborts on any disagreement') and the registered gate onFail 'abort' imply per-reply enforcement, and the run is long (hours, resumable). If the seat substitutes the model mid-run (quota routing, rollout), every later step is served by a model nobody registered and the run continues; the report's servedModels list only records the mix after the fact. Since rows are persisted and the run resumes, a mid-run substitution poisons all subsequent rows. Consider aborting on the first step whose servedModel fails the registered gate (the gate is already registered; per-step evidence is available in StepRecord.servedModel).

🟡 LOW identity-abort artifact misreports spend: probe tokens are nonzero but spentUsd is hardcoded 0 — scripts/tb-gated-stop-ab.ts

The refusal written to confirm-refusal.json sets spentUsd: 0 and the note claims 'No row was graded and no arm was run' and the stdout claims 'no spend'. But the identity probe (callModel, line 1232) is a real seat call that consumed tokens before the gate failed. The committed artifact (benchmarks/trace-repair/gated-stop-ab/confirm-refusal.json) carries the same zero while the probe billed usage. Either count the probe's usage in spentUsd or phrase the note as 'no row spend' — as written, the artifact understates the true abort cost.

🟡 LOW registered pairedContrast95 interval measures pooled pass rate, not the paired contrast — scripts/tb-gated-stop-ab.ts

registered.interval('pairedContrast95') executes the registered cluster-bootstrap (computeInterval) which resamples clusters of per-row passed values pooled across BOTH arms. That interval bounds the overall pass rate, not the treatment-control difference the pairedContrast estimand computes. The decision branches on interval-excludes-zero sign are therefore uninformative about whether gated-stop beats blind continuation. The interval spec predates this PR (unchanged in the diff), but the new executor is where it materializes into the report and where it feeds the (currently unexecuted) decision. Either register a genuinely paired interval (per-pair contrast values, resampled by cluster) or relabel the artifact so its semantics are not read as a contrast.

🟡 LOW settlingDraw returns scanned with maxRows appended after the sort — scripts/tb-gated-stop-ab.ts

scanned.sort is applied at line 380, but the returned object then calls evaluate(maxTake) (line 389). When no coarse-grid step reached maxTake (the common case where a draw clears before the ceiling), evaluate(maxTake) pushes a new entry after the sort, so the auditable scanned curve is not fully sorted despite the comment at line 324. Cosmetic; sort before returning.

🟡 LOW settlingDraw search cost: ~27 powerAt calls at 15k x 3k x ~22 clusters each — scripts/tb-gated-stop-ab.ts

The committed design.json scanned 27 points; each powerAt call runs 15,000 trials x 3,000 bootstrap resamples x clusterCount (~22) RNG operations, plus a 15k-element sort per call — on the order of 10^9-10^10 ops per call, tens of billions overall. Design mode took over an hour by wall-clock (committed generatedAt 12:02 -> 13:20 includes the search). This runs on every design and every confirm invocation (design section executes before mode dispatch), and also when --take is passed even though settling is then unused. Acceptable as one-time design cost, but memoizing at a coarser rows grid or skipping the search under --take would cut the design loop materially.

---

_{tangletools · 2026-08-14T14:29:28Z · trace}

[tangletools]

❌ 3 Blocking Findings — c108ed32

Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 4/4 planned shots over 5 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-08-14T14:29:28Z · immutable trace}

[drewstone]

Findings disposition for c108ed32, addressed in 96089e2 (branch rebased onto current main):

1. CRITICAL boolean passed crashes the registered contrast — FIXED. Reproduced: computeEstimand throws paired-mean-diff value field 'passed' is not a number on the boolean shape, and computes {value, numerator, denominator} on the numeric shape (executed against src/experiment/ast.ts directly). armRows now writes passed: passedOf(run) ? 1 : 0 (scripts/tb-gated-stop-ab.ts:1343-1352).
2. HIGH refuse-spend halt never enforced in confirm — FIXED. Confirm now checks halt.fired before the identity probe and before any spend; a fired halt writes confirm-refusal.json with the gate result and exits code 4 with zero rows graded (scripts/tb-gated-stop-ab.ts:1228-1250).
3. HIGH confirm always crashes on the contrast — same root cause as finding 1. One defect, one fix; see item 1.

Also updated: the PR body and docs/trace-repair-gated-stop.md now record the completed run's corrected result (302 rows, +0.0596, 95% CI [-0.0061, +0.1210], includes zero, 15 gated-arm rows degraded by 429s) instead of standing as a pre-run prediction.

Proof: tsc --noEmit -p tsconfig.script.json clean; vitest run tests/experiment src/experiment 62/62 passed.

@tangletools review now

[tangletools]

✅ Auto-approved drewstone PR — 96089e22

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-15T10:09:22Z}