feat(p1-parity): honest live entry — deterministic task family, harness oracle, driver protocol - #869
Conversation
…rity harness The live entry gets a deterministic count-and-positions task family whose ground truth the harness computes, so the deliverable check is a deployable text oracle identical in both arms. The reviewer profile carries the graph driver protocol in prompt.instructions; the shared systemPrompt stays the multishot driver stance. A per-run nonce keeps prompts unique across runs so an upstream cache cannot cross-serve completions. The chat backend gains a runGraph maxTurns bound, and the CLI prints a per-cell parity summary with an edge-ledger accounting audit.
|
@tangletools review now |
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 380dfc28
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-15T10:44:53Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 380dfc28
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-15T10:45:01Z
|
@tangletools review now |
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 380dfc28
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-15T10:50:12Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Coverage | 2 of 2 lenses (value, usefulness) |
| Concerns | 3 (1 medium-concern, 1 low, 1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.3s |
| Interrogation | 174.5s (2 bridge agents) |
| Total | 174.8s |
💰 Value — sound-with-nits
Replaces the live entry's self-graded 'ALL TESTS PASS' marker with a harness-computed exact-match oracle plus a real driver protocol for the graph arm — a genuine measurement-validity upgrade in the codebase's grain; the only material gap is that the fixed famous-word task family, per the PR's own l
- What it does: Three deltas in the p1-parity live entry (examples/p1-parity/run-parity.ts): (1) LIVE_TASK_FAMILY — 8 fixed count-and-positions cells ('strawberry'/r, etc.) whose ground truth expectedAnswerLine() computes in harness code, checked by liveShotPassed() demanding exactly one ANSWER: line byte-equal to it — replacing the old LIVE_PASS_MARKER where the model itself decided when to print 'ALL TESTS PASS
- Goals it achieves: Makes the live parity measurement honest. The old marker oracle let the coder self-certify, so a 'converged' row measured nothing; now ground truth is computed by the harness and the oracle is shared byte-identically by both arms (the MultishotArmBackend.shotPassed contract at arms.ts:171-173 explicitly requires the same predicate as the graph deliverable). The driver protocol gives the graph arm'
- Assessment: Good, and in the grain. The design reuses the right primitives instead of inventing: spawn_agent/await_event match the kernel's actual coordination tools (examples/graphs/shot-loop.ts:71-91), maxTurns is a real RunGraphOptions field, instructions ride the existing profile field with the documented append semantics, and the ledger audit reads the same EdgeTraversal rows ParityRecord already exposes
- Better / existing approach: Searched for existing reusable oracle/task-family machinery: bench (LLM-judge fixtures, corpus-replay), examples/coding-benchmark (held-out test-execution oracle), examples/agentic-data-creation (rubric data-gen), src/ (no ANSWER-style exact oracles), and the agent-eval multishot surface (transport/driver seams only, no verifier lens). None fits the bare-chat no-tools no-third-model constraint, so
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound
Replaces the live entry's self-graded marker oracle with a deterministic harness-computed oracle and a reviewer driver protocol that ride existing runtime knobs — the change is coherent, correctly wired, and already exercised by a full live run.
- Integration: Reachable and already used. The entry is the documented CLI (
pnpm tsx examples/p1-parity/run-parity.ts --backend cli-bridge, run-parity.ts:7-11), matching the repo-wide examples convention (examples/README.md:28-30,136-139). The live path is deliberately outside CI gates (header, run-parity.ts:17) while the offline suite still exercises both arms via the seams (parity.test.ts:103-326). Every new - Fit with existing patterns: Fits the grain rather than competing. The counting family + harness-computed ground truth implements the doctrine the file itself states (run-parity.ts:113-118: oracle read off the coder's text, never the model judging itself) and directly retires the previous placeholder — the deleted
LIVE_PASS_MARKER = 'ALL TESTS PASS'includes-check was self-graded and explicitly labeled 'pending a real verif - Real-world viability: Built for the non-happy path. The oracle demands exactly ONE matching ANSWER line (run-parity.ts:148-156), so shotgunned candidates or prose cannot pass; ground truth is loop-computed, never model-claimed. Per-run nonce (run-parity.ts:397) defeats upstream response caches while preserving within-cell input equivalence (same nonce reaches both arms). Missing env fails loud with a complete list (run
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
🔎 Heuristic Signals
🟡 Cruft: console debug added examples/p1-parity/run-parity.ts
- console.log('\nparity summary (per cell):')
💰 Value Audit
🟠 Task family never exercises the multi-shot path with the tested model — the regime P1 measures [better-architecture] ``
run-parity.ts:113-118 claims 'shot 1 fails often enough to exercise the multi-shot path'. The PR's own live table contradicts this: all 5 visible graph-arm rows report shotsUsed=1, steering 0x/0B — the deliverable passed on the first spawn every time, so the graph arm's re-brief/steer/early-stop machinery and the arms' divergence behavior were never live-tested. Famous-word cells ('strawberry' most of all) are prime memorization candidates, which plausibly explains uniform shot-1 passes for glm-
🟡 Duplicated task-family bounds guard [maintenance] ``
The identical LIVE_TASK_FAMILY[index] undefined check exists at run-parity.ts:174-179 (inside liveParityCell) and run-parity.ts:411-416 (in main, needed to compute expectedAnswerLine before building the cell). liveParityCell's copy is unreachable from its only caller since main pre-checks. A generated family with no fixed cap dissolves this; otherwise drop the inner guard or have liveParityCell return the task alongside the cell.
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
|
Disposition on the value-audit concerns (verdict sound-with-nits, head 380dfc2):
|
2 participants
The P1 parity harness gets its honest live entry: a deterministic count-and-positions task family with harness-computed ground truth, an exact-match text oracle shared by both arms, and the reviewer driver protocol for the graph arm. Nothing in the repo's gates executes the live path; the offline suite still covers the arms.
Part of #694 (P1: loop-vs-graph parity).
What changed
examples/p1-parity/run-parity.ts—LIVE_TASK_FAMILY(8 deterministic cells),expectedAnswerLine(ground truth computed by the harness, never by a model),liveShotPassed(exactly one matchingANSWER:line), live cell authoring (coder + reviewer profiles from env, driver tool protocol ininstructions), per-run nonce so an upstream response cache cannot serve one run's completion to another.examples/p1-parity/arms.ts— doc corrections on the record fields the live entry exercises.Gates
pnpm typecheck— clean (root + examples).pnpm vitest run examples/p1-parity/parity.test.ts— 5/5 passed.origin/main— clean.Live run evidence (8 cells x 3 shots x 2 arms, glm-5.3 both arms/legs, served-model verified)
usd @router = tokens priced at the router's own response headers for glm-5.3 ($1.68/M input, $5.28/M output). The records' own
usdfields are labeled byusdSource(multishotestimatedfrom a stale catalog, graphunknown— it never estimates); neither is a router-header measurement, so the table prices tokens uniformly.Verdict: parity HOLDS. Convergence agrees in 8/8 cells,
infraShots= 0 in 16/16 records,tokensKnown= true in 16/16, and the graph edge ledger accounts in 8/8 cells (every delivered delegates row bound to a worker; distinct workers = shotsUsed; shot budget respected). Zero diverging cells.Shot distributions: multishot 3,3,3,3,3,3,3,3 — the full budget in every cell, BY CONSTRUCTION (
runMultishothas no deliverable check and cannot stop early). Graph 1,1,1,1,1,1,1,1 — settled on the first shot in 8/8 cells (the smoke's 1-shot pattern repeated).The known design asymmetries, measured
valid:truefrom the harness oracle, so the driver stopped after spawn fix: persist final runtime stream failures #1 in 8/8 cells: 0 steers, 0 bytes. The multishot driver leg never sees a verdict, so it delivered 2 corrective steers in every cell — 16 steers, 2732 bytes total — and the loop burned 24 shots for 8 convergences.buildOpenerreturns the task byte-exact. The graph driver copies the task intospawn_agentper its protocol, and the edge ledger measures the fidelity: delivered brief bytes minus task-text bytes = 424 bytes in all 8 cells — a constantdelegates/worker-brief/v1directive wrap over a byte-exact copy. Zero copy drift, and the exact-match oracle passing on shot 1 confirms it independently.One cost asymmetry to keep visible: the graph arm is input-heavy (~13.4k tokens/cell of supervisor context vs ~1.7k for the loop), so at router prices it cost more per cell ($0.0261 avg vs $0.0171) despite using 1 shot vs 3 — while finishing faster in 7/8 cells (median wall 44.7s vs 82.8s).
Full artifacts (paired records JSON, ledger rows) on the #694 comment.