Prune the test lanes that don't pay for themselves, and add the checks we're missing

[thymikee]

Follow-up to #1412. CI costs about 100k billed minutes a month, mostly on macOS runners. Some lanes are not paying for themselves, and a few kinds of bugs have no test looking for them at all.

The rule

Every test lane answers four questions in writing before it lands, and keeps answering them:

1. What bug does it catch that nothing else catches?
2. What has it caught, or which real bug made us want it?
3. What does it cost — harness code, minutes a month, time added to a PR?
4. When would we delete it?

No answers, no lane. Delete condition met, lane goes. Three notes so the rule doesn't delete the wrong things: a cheap lane guarding code that is currently correct is fine — "caught nothing" is not "worth nothing" when it costs under a minute; a preventive guarantee (a layering rule, a coverage floor on new lines) is removed only when something else enforces the same thing or the thing it guards is gone — never because it was quiet; and a nightly that is just a bigger version of a PR lane has to say what the extra runs found.

We also start reporting how much test-harness code we own (about 25k lines, ~15% of product code) next to the size report.

Prune or fix what we already run

• A1 — replays-nightly.yml full-tier jobs. cost · wave 1 Red 25 nights in a row; Android fails the same permission assertion every night, still after fix: stabilize Replay Nightly fixture boundaries #1610. The lane is doing its job loudly — nobody owns the fix. Fix within a week, or switch the jobs off until someone does. "Manual" means off; that's fine. (merged: fix test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793 + park ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) #1794; Android full tier 9/9 on the lane; replay-android overlay window remains, tracked separately)
• A2 — nightly-macos job. (merged: ci: drop duplicate nightly-macos replay job (#1781 A2) #1787) cost · wave 1 Byte-identical to macos.yml, which already ran on the same commit. Delete it.
• A3 — perf-nightly.yml. cost · wave 2 (decided) Writes a report and never compares it to anything, so it can't notice a slowdown (it fails only if the harness crashes). Add a comparison, or turn it off. (decided+merged: ci: park perf-nightly to dispatch and stop the coverage-gate cascade double-red (#1781 A3, A5) #1822 — parked to workflow_dispatch; iOS medians swing up to +122% night-to-night at n=5, no consumer; un-park when a smoothed comparison exists and someone owns reading it)
• A4 — contention-retry and the subprocess-stub project. harness · wave 2 (decided) Two things, decide them separately. The stub project serializes 15 tests that spawn binaries or touch process-global state; the retry allows one narrowly classified runner-timeout retry and refuses everything else, and each entry already carries an expiry date. Its tracker Slow-test ratchet: convert real-time waits to injected budgets #1098 is closed. Per entry: can isolation, fake clocks, or an injected retry policy remove the real wait? Convert those; keep the rest with a live tracker. (decided+merged: chore(test): remove the contention retry (0 fires since 07-29) and shrink the subprocess-stub project to its 3 real spawners (#1781 A4) #1827 — retry deleted (0 fires since 07-29), stub project shrunk to 3 real spawners; tracker subprocess-stub vitest project: members and kill criterion (#1781 A4) #1823; side finding Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824)
• A5 — check:coverage-changed. harness · wave 2 (measured, kept) Guards a different thing than the global threshold: 70% coverage on the lines a PR adds, so a green global number can't hide an untested change. Keep the guarantee. Log blocks and waivers for 60 days to judge its noise, and merge its execution into the coverage job if the two duplicate work. (measured retrospectively over 60 days instead: 1 legitimate block (feat(ai-sdk): add agent-device/ai-sdk tool set and document the MCP zero-code path #1804 at 6.9%), 0 waivers, 16 of 17 reds were if: always() cascades — fixed in ci: park perf-nightly to dispatch and stop the coverage-gate cascade double-red (#1781 A3, A5) #1822; execution was already inside the coverage job)
• A6 — check:layering. harness · wave 2 (reviewed) Our only structural gate, 13 policies. For each: which invariant and ADR/CONTEXT section it enforces, whether TypeScript or package exports already make the violation impossible, and a planted-violation proof that it still fires. Retire a policy only when it is redundant, type-enforced, or its invariant is gone — not because it was quiet. (reviewed+merged: chore(layering): document R12/R18/R19, retire R8, make R9 shrink mandatory (#1781 A6) #1825 — kept in full, 15/15 planted violations fired; R12/R18/R19 documented, R8 retired (no subjects), R9 shrink mandatory, cutover expiry in ADR 0019)
• A7 — iOS XCTest. harness · wave 1 (run) → wave 3 (classify) CI runs 37 of 153 tests (a hand-written -only-testing list); the other 116 run nowhere. Run the full suite nightly and add a check that the list can't silently drop tests. Then sort the 153: pure Swift decision tests → a fast unit target; runner/XCTest semantics → keep on the simulator; behaviour the CLI smoke already observes, or stale → delete. (run half merged: ci(ios): run the full XCTest suite nightly and check the PR test list (#1781 A7) #1789 — nightly full suite + xctest-selection gate; first execution found ios runner: navigation fallback helpers accept CGRect.infinite (tap point ≈ −9e307) #1812; classify half stays open)
• A8 — concurrency torture nightly. cost · keep Green, cheap, never found a bug. Keep the fast PR sweep; keep the nightly under the cheap-lane note above.
• A9 — ios.yml. Over half our spend; it is the only thing testing the runner, so it can't be cut — only started less often. Measured twice (comments below). Cancel rate is ~19% over 995 runs and a cancelled run burns a median 6.5 min first, so cancel waste is real. Two separate findings: (1) ship now — paths-ignore on ios/android/linux/macos(/ci) skips docs/** and README.md but not CONTEXT.md, AGENTS.md, SECURITY.md, CHANGELOG.md; three docs-only PRs (docs: add security policy #1568, docs: adopt request-bound platform runtime #1697, docs: prefer design fixes over regression-only guards #1722) each ran a 9–15 min iOS job. Add those files to the ignore lists. (2) routing is under-tooled, not disproven — a 150-PR sample finds ~10% touching no Apple-or-shared path, but the affected selector has no ownership rule for replay-ios at all and assigns swift-runner-ios only for .swift/apple/runner/**, missing TS-only Apple PRs. Give the model real Apple ownership rules first, then re-measure over 90 days by file classification. cost · (1) wave 1 · (2) wave 3 (A9-1 merged: ci: skip device lanes for root-level docs-only changes (#1781 A9) #1791; A9-2 routing on hold)

Not reviewed, cheap by construction: size, build cache, deploy/preview/release wrappers, conformance lanes.

Add

• B1 — daemon leak check. observers · wave 3 (after B4) After a session closes and after shutdown: no daemon-owned child processes beyond the declared baseline, no request/session resources left, nothing unexpected in an isolated state dir. Owned resources, not global process counts — simulator and XCTest own things we don't. Both leaks we know about (web: agent-browser Chrome fleets leak and can spin at 100% CPU for days — add idle lifecycle/reaper #1109, iOS simulator recorder leaks on session/daemon teardown while recording is active (0-byte recording.mp4, "Host recording is already in progress") #1324) were found by hand; the check must go red against their pre-fix behaviour. Around the smoke tests, no added minutes.
• B2 — fuzzing v2. observers · any time Give CLI args and Maestro commands real input generators (today their mutations die in the tokenizer) and assert the specific error code, not "some error". Done when generators reach past the tokenizer into command validation, the two known bugs (fix(cli): reject excess positionals #1433, the replay parse leak) are rediscovered from a clean corpus, and nightly runtime doesn't grow. Stands on its own.
• B3 — fuzzing calibration. observers · wave 3 (after B2) Measure whether the fuzzer can rediscover bugs we already have on file. Nothing measures this, so we don't know if the nightly run does anything. Small, bounded experiment; if it can't rediscover a seeded set within the budget, it doesn't go into CI.
• B4 — assert the right reason. (merged: test: assert the specific error code instead of any failure (#1781 B4) #1790; follow-up utils: app-log-files and verified-file throw plain Error instead of AppError #1792) observers · wave 1 21 assertions accept any failure, not the one they name. Fix them, add sync/async helpers that assert code (and details.reason where it matters), allow explicit exemptions where the contract really is "propagate this exact error". No broad "is this assertion specific enough" scanner.
• B5 — tiny limits. observers · last Make a few hard-coded caps overridable so tests can actually reach them (fix: add iOS private AX snapshot fallback #758 was this class). And events.ndjson has no rotation cap — that's a bug, file it on its own.

Not doing

A sanitizer XCTest lane — no evidence of native races to catch; run it once by hand after A7 and see. Coverage-guided fuzzing as its own lane. HWASan/MTE, E-core budgets, new LLM-judge suites. Details in the first comment.

Do first

Wave 2 CLOSED 2026-08-18: #1822 (A3+A5), #1827 (A4), #1825 (A6), #1828 (#1457 → report-only) all merged; decisions in the wave-2 comment; side issues #1823 #1824 #1837. Wave 3 handoff in the comment below.

Wave 1 (2026-08-17): A2 merged (#1787); A1 fix #1793 + park #1794, A9-1 #1791, B4 #1790 ready for review; A7 #1789 draft pending the dispatch decision; A9 measured (routing on hold); events.ndjson → #1788; follow-ups #1792, #1796. Boxes get ticked as PRs merge.

1. Delete A2 — one line.
2. Fix or switch off A1 — both merged (test(android): restore full-tier lifecycle and observability scenarios (#1781 A1) #1793, ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) #1794).
3. Run the A9 measurement — done twice; ship the paths-ignore fix now (A9-1), hold routing (A9-2) until the selector can express Apple ownership.
4. Get the 116 XCTests running somewhere (A7) — merged ci(ios): run the full XCTest suite nightly and check the PR test list (#1781 A7) #1789; the classify/delete pass is wave 3.
5. Do B4 (merged test: assert the specific error code instead of any failure (#1781 B4) #1790), then the scoped B1.
6. Decide A3; audit A4 per entry — merged (ci: park perf-nightly to dispatch and stop the coverage-gate cascade double-red (#1781 A3, A5) #1822, chore(test): remove the contention retry (0 fires since 07-29) and shrink the subprocess-stub project to its 3 real spawners (#1781 A4) #1827).
7. Instrument A5/A6 — measured and kept (ci: park perf-nightly to dispatch and stop the coverage-gate cascade double-red (#1781 A3, A5) #1822, chore(layering): document R12/R18/R19, retire R8, make R9 shrink mandatory (#1781 A6) #1825).
8. File the events.ndjson rotation bug — daemon: events.ndjson has no size/rotation cap #1788.
9. Wave 3 (open): B1 → test: deterministic boundary fault-injection matrix for daemon HTTP and subprocess seams #1431 · A7-classify · A9-2 routing · B2 → B3 · B5 (+daemon: events.ndjson has no size/rotation cap #1788) · changed-line mutation experiment (from gate(mutation): ratchet graduates on stability alone — a 28.75% score can lock in as the permanently gated floor #1457) · Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824 Coverage fork death · check:layering R10 names the wrong file when a zone exceeds its type-cycle ceiling #1837 R10 message.

Carried forward from #1412: #1431 (pairs with B1), #1457 (closed by #1828: shrunk to report-only), #1481, #1584 (blocked on A1), #1426 (rescope to the non-Android platforms).

Views: Test Atlas · Test Topology

[thymikee]

Revised (2026-08-17): every Part A/B item now carries an explicit expected-value statement — Catches / Evidence / Cost / Kill criterion — and the umbrella adds that as a standing rule: a lane that can't fill in all four lines doesn't land, and one that stops meeting its kill criterion is deleted. Applying it dropped the sanitized-XCTest lane (no evidence line; run once as an experiment after A7 instead), folded coverage-guided fuzzing into B3 as the value-capture instrument for the fuzz lane, narrowed tiny-limits, and added A8 (concurrency-torture) since the rule has to apply to green lanes too.

[thymikee]

Second revision (2026-08-17), from reading the suite as shapes rather than tables (Atlas + Topology views now linked in the body):

• A9 added — ios.yml is >55% of spend and its cancel waste alone (~13k min/mo) equals A1+A2+A3 combined; it runs on every non-docs push regardless of what the diff touched. Sole observer of the 19k-LOC runner, so it can only be made cheaper per run: measure, then route through the existing affected selector.
• Two corollaries added to the rule — (1) every nightly is a scale-up of a PR lane, not a different observer, so each must show what its extra ring found (B3's calibration becomes the nightly column's instrument); (2) test infrastructure is ~25.5k LOC / ~15% of product and gets a tracked budget line.
• A1/A3 wording — 'demote to manual' is deletion in practice; said so.
• Reading that also sharpened the priority argument for B1/test: deterministic boundary fault-injection matrix for daemon HTTP and subprocess seams #1431: the daemon is dense in test count and thin in observer kind — all six unobserved defect classes are daemon-shaped.

[thymikee]

Provenance (moved out of the body to keep it short). An outside cold review found the previous body ~2,500 words holding four documents; this is the plain-language version. Corrections it forced: A3 wording (a harness crash does fail the lane — the point is that nothing is compared); A1 reframed (the lane is loudly correct, the fix is unowned); A7's 'green 90 days → delete' replaced with an overlap test; the rule now exempts cheap lanes so it doesn't delete correct regression barriers; A5/A6 criteria are forward-looking tallies (the counterfactual 'name a PR it would have blocked' can't be computed); B2 no longer depends on B3.

Verified evidence behind the items (2026-08-17, HEAD 856ff38):

• Cost: ≈100k billed-min/mo estimated from the last 40 runs per workflow (runs/mo × median duration × macOS 10×). ios.yml ~59k, 22% cancelled. Estimates — A9 starts by measuring.
• A1: 25/25 scheduled runs red since 07-24; Android assertion request Android microphone permission for deny unchanged after PR fix: stabilize Replay Nightly fixture boundaries #1610 (merged 08-05).
• A2: replays-nightly.yml nightly-macos = macos.yml (macos-helper + replay-macos --retries 2), 0/40 failures.
• A3: scripts/perf/run.ts writes a report; no threshold or baseline comparison anywhere in scripts/perf.
• A4: CONTENTION_RETRY_FILES (15 files) cites Slow-test ratchet: convert real-time waits to injected budgets #1098 as tracker; Slow-test ratchet: convert real-time waits to injected budgets #1098 closed COMPLETED 2026-07-07.
• A7: 37 -only-testing: entries in ios.yml vs 153 func test methods under apple/runner.
• B1: no FD/child/timer census anywhere; /health reports version only; only run-level tmpdir scan exists.
• B2: cli-args and maestro have no entry in STRUCTURED_BASES (scripts/fuzz/arbitraries.ts); vocabularies exist (listCliCommandNames(), SUPPORTED_MAESTRO_COMMAND_NAMES); kernel has 17 closed error codes.
• B4: 21 matcher-less assertions (4 toThrow(), 1 assert.throws(fn), 16 assert.rejects(p)); assertRejectsAppError used in 7 files, no sync sibling, none in packages/*.
• Harness LOC ≈25.5k: layering 10k, mutation 2.4k, contention 1.7k, fuzz 1.7k, check-affected 1.6k, wire-compat 1.5k, depgraph 1.5k, output-economy 1.4k, gate 1.2k, perf 0.9k, coverage-changed 0.8k, replay-compat 0.6k.

Not doing, with reasons: sanitizers — runner is ARC Swift with ~40 objc_msgSend/NSSelectorFromString sites, zero native race/UAF fixes in 300 commits, macOS helper has no test target, Android helper is Java (HWASan n/a); run TSan/ASan once by hand after A7. Coverage-guided fuzzing — Jazzer.js is a poor fit (ESM/strip-types/worker watchdog); if B3's small experiment shows the nightly finds nothing, that's the answer. E-core PMU budgets — laptop CLI, not phone silicon. New LLM-judge suites and scheduled telemetry without a consumer — #1412's descope stands.

Split option: if this still reads as too much for one issue, the natural cut is three — cost prune (A1, A2, A3, A9), harness-debt decisions (A4–A7), new checks (B1–B5).

[thymikee]

Third revision, after a second independent review whose central point I accept: recent silence is not evidence that a preventive guarantee has no value. Changes in the body:

• The rule now says preventive guarantees (layering rules, the added-lines coverage floor) are removed only when redundant, type-enforced, or guarding something that no longer exists — never for being quiet.
• A5 — I had this wrong: check:coverage-changed gates 70% on the lines a PR adds (scripts/coverage-changed/model.ts), which the global threshold cannot guarantee. Keep the guarantee; judge noise (blocks + waivers) and merge execution if duplicated.
• A4 — split into two decisions (stub project vs. retry); entries already carry expiry dates that force review; convert per entry, don't delete wholesale.
• A6 — per-policy: invariant + ADR, is it type-enforced elsewhere, planted-violation proof; retire on redundancy only.
• A7 — classify the 153 (pure Swift → unit target; XCTest semantics → simulator; CLI-observed or stale → delete) and add a discovery check so the -only-testing list can't silently drop tests.
• A9 — needs a routing design: decide before allocating macOS, fail-open on tooling changes, transitive Apple ownership, push-to-main always, retrospective replay against recent PRs before enabling.
• B1 — scoped to daemon-owned resources with a red proof against web: agent-browser Chrome fleets leak and can spin at 100% CPU for days — add idle lifecycle/reaper #1109/iOS simulator recorder leaks on session/daemon teardown while recording is active (0-byte recording.mp4, "Host recording is already in progress") #1324 pre-fix behaviour, not global process counts.
• B4 — helpers + explicit exemptions; no broad assertion-specificity scanner.
• B2/B3 — acceptance and stop conditions stated.

Body is ~1,150 words. Both reviewers recommend splitting into three children (cost & scheduling: A1–A3, A8, A9 · harness ownership: A4–A7 · missing observers: B1–B5) with this issue holding an outcome table. Not done yet — maintainer's call on whether one issue or three is simpler to work.