docs: restructure the README around front doors, and read a binary outcome as 1 or 0 by drewstone · Pull Request #613 · tangle-network/agent-eval

drewstone · 2026-08-15T09:17:18Z

What changed

The README carried 238 of its 507 lines on one API family (external text optimizers), and 8 of its 10 TypeScript blocks did not compile. concepts.md described selfImprove() as returning a ship/hold decision when the real type has five values, and its only TypeScript block was a parse error.

README: 507 → 190 lines. One quickstart, then a table with one row per callable front door.

Column	Content
When to call it	The situation, in the reader's words
What you give it	The inputs
What you get back	The result

Eighteen rows, including the surfaces that had no README presence at all: /experiment, analyst definitions, verification strategies, trace-repair, and trajectory-replay. Every row links to a runnable example or the doc that owns it.

Examples: 7 new directories

Each holds one runnable file and a README that answers WHEN / HOW / WHY. Every expected output printed in a README is the output that file produces.

Directory	When to use it
`evaluate-a-change`	You changed a surface and must know whether it helped
`plan-before-you-spend`	The run costs money and you must see the grid first
`analyze-existing-runs`	The runs already happened; no agent needs to run again
`sealed-experiment`	The result must convince someone who does not trust you
`verify-without-an-answer-key`	The work has no held-out test suite
`custom-trace-analyst`	A batch failed and you need cited findings
`adapt-a-text-optimizer`	Another package owns the search, you keep the scoring

All 7 typecheck under pnpm typecheck:examples and all 7 run offline.

Fix: `computeEstimand` rejected a binary outcome

A registered binary outcome reaches evidence as true/false. computeEstimand threw on that type, so a paired-mean-diff over a pass/fail field could not run without the caller re-encoding every row — which is exactly the shape scripts/tb-gated-stop-ab.ts registers (value: 'passed'). computeInterval rejected it the same way.

One shared reader now gives both interpreters the same reading:

a boolean reads as 1 or 0, so the mean of paired differences is the risk difference;
a finite number passes through;
every other type, and a non-finite number, rejects loudly.

NaN previously passed the typeof === 'number' check and silently poisoned the mean. It no longer can.

tests/experiment/paired-binary-outcome.test.ts runs the gated-stop runner's exact spec shape end to end: seal → verify → admit → estimate → interval → decide, on boolean rows. It asserts the risk difference (9/16), that hand-encoded 1/0 rows give an identical result, that zero-diff semantics survive a missing arm, and that five bad value types reject. 10 of 10 tests fail without the fix.

Docs

Python install commands and version pins deleted from the README; the optimizer guide already held them. Added the release-vs-source engine note there.
Engine list corrected: it named AutoResearch, Meta Harness, and Best-of-N, none of which this package implements, and listed best-of twice. The real recipes are engine, sequential, adaptive-sequential, best-of, vote, omni.
concepts.md: the five GateDecision values with what each one tells you to do next; three new front-door rows; a vocabulary section covering the terms the docs use and never defined (case, surface, dispatch, campaign, cell, receipt, cost ledger, provenance, train/selection/final cases, seal, estimand, funnel, certification); the judge-calibration statistics as a table instead of a 66-word sentence of unglossed symbols.
feature-guide.md: dropped a row naming runPromptEvolution, which does not exist.
verify-package-exports.mjs follows the renamed quickstart heading, so the packed-build check keeps running.

Proof

Check	Result
`pnpm typecheck`	clean
`pnpm typecheck:examples`	clean
`pnpm build`	clean
`pnpm verify:package`	exit 0 — compiles the README quickstart against the packed build
README TypeScript blocks	2 of 2 compile (was 2 of 10)
New examples	7 of 7 typecheck, 7 of 7 run
Relative links across README, docs, examples	0 broken
`pnpm test`	5257 passed, 1 failed

The single failure is tests/contract-self-improve.test.ts > threads rationale + diff + durable provenance. It is a pre-existing flake: it fails intermittently on clean origin/main too (fails run 1, passes run 2, same checkout), and it has no import path to computeEstimand. Not introduced here, not fixed here.

A registered `binary` outcome reaches evidence as `true` or `false`. `computeEstimand` rejected that type, so a paired-mean-diff over a pass/fail field could not run without the caller re-encoding every row. `computeInterval` rejected it the same way. One shared reader now gives both interpreters the same reading: a boolean is 1 or 0, a finite number passes through, and every other type or a non-finite number rejects. A `NaN` outcome no longer reaches the mean.

…seal, and verify Each directory holds one runnable file and a README that answers when to use it, how to run it, and why it is built that way. Every expected output in a README is the output the file prints.

…aims The README carried 238 of 507 lines on one API family and 8 of 10 TypeScript blocks that did not compile. It now leads with one quickstart that the package verifier compiles against the packed build, then a table with one row per callable front door and a link to a runnable example for each. The Python install commands and version pins move to the optimizer guide that already held them. The engine list drops three names the package does not implement and no longer lists best-of twice. concepts.md states the five release decisions, not a ship/hold pair; its only TypeScript block now parses; and it defines the vocabulary the rest of the docs use. feature-guide.md drops a row naming a function that no longer exists.

tangletools