feat(supervise): acknowledged worker cancellation via the run layout - #854
Conversation
Closes #758. The run layout gains a worker-scoped cancellation contract: - cancelWorker appends a durable request keyed by a caller-minted operationId and returns the operation's current durable state. Idempotency is a lookup: an existing acknowledgement returns as-is, so a repeated operation can never apply twice. - The acknowledgement record reuses RetainedRunEffect. The coordination driver's turn loop is the acknowledger: one cancellation-inbox read per turn, applied through the worker's existing per-child abort chain. It records cancel_requested when the abort is issued, cancelled only when the worker reaches a terminal down state on the settle path, and not_live when the worker is already gone. A missing worker never reads as success. - A cancelled lead cascades to its subtree by design; the record's terminated set names every node id proven terminated, filled from the durable spawn journal as descendants settle. - supervise() wires runDir as the control surface; the TUI's worker-focused cancel now goes through cancelWorker. The run-level cancel stays a host-honored file because a non-retained tree has no root handle to acknowledge with (a separate issue). Runtime tests cover the acceptance list: idempotent repeat, sibling isolation, descendant naming with a surviving sibling subtree, reconnect read from the layout alone, not_live/unknown for a gone or unknown worker, and the terminal state on the settle path.
|
Full local |
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 42a6bda4
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-15T03:27:34Z
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 42a6bda4
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.
tangletools · auto-approval · reason: drewstone_author · 2026-08-15T03:29:14Z
tangletools
left a comment
There was a problem hiding this comment.
⚠️ Value Audit did not run — no verdict
This is not an approval and not a clean bill of health. Neither interrogation lens returned a judgment, so this PR has no value-audit evidence for or against it.
| Status | audit-incomplete (could not run) |
| Why | value: cli-bridge admission rejected (queue saturated); usefulness: cli-bridge admission rejected (queue saturated) |
| Lenses answered | 0 of 2 |
| What to do | re-run once the CLI bridge has capacity: pr-reviewerctl trigger <repo>#<pr> --force |
💰 Value — error
value agent never ran: the CLI bridge refused admission (no model was started).
- Model: opencode/deepseek/deepseek-v4-pro
- Bridge attempts: 4
- Bridge error: opencode/kimi-for-coding/k2p7: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=3/48 — no model was started
🎯 Usefulness — error
usefulness agent never ran: the CLI bridge refused admission (no model was started).
- Model: opencode/deepseek/deepseek-v4-pro
- Bridge attempts: 4
- Bridge error: opencode/zai-coding-plan/glm-5.2: Bridge returned 503: bridge at capacity (queue_timeout, lane=reserved): active=20/20 queued=4/48 — no model was started
🔎 Heuristic Signals
🟡 Cruft: magic number added tests/kernel/worker-cancellation.test.ts
+const perWorker: Budget = { maxIterations: 4, maxTokens: 1000 }
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
Adversarial review (post-merge — the value audit that published 'no verdict')Read against Confirmed defect — fixed in #859The TUI offered a worker cancel that can never be acknowledged. Deferred with evidence
Claims that survived refutation attempts
Checks: |
Closes #758. Implements the audited design in #758 (comment).
What ships
1. Acknowledgement record in the run layout (
src/runtime/supervise/run-layout.ts)WorkerCancelRequest— one durable NDJSON line incancellations/requests.ndjson, keyed by a caller-mintedoperationId.WorkerCancellation— the durable acknowledgement atcancellations/<opId>.json, written ONLY by the runtime acknowledger.effectreusesRetainedRunEffect(cancel_requested | cancelled | not_live | unknown) — no second four-state vocabulary.operationIdis returned as-is; a repeated operation never applies twice. Filename sanitization collisions between distinct operation ids fail loud on read.2. The acknowledger in the existing turn loop (
src/runtime/supervise/coordination-driver.ts)DriverAgentOptions.controlDirmounts the acknowledger insidehooks.beforeTurn— one cancellation-inbox read per turn, plus one final pass after the post-loop drain. No new process, no poller, no socket, no new lifetime.CoordinationTools.abortWorkerkeeps theHandlefrom the spawn that minted it and callshandle.abort(), which cascades to that child's subtree and no sibling (the scope's ownAbortControllerchain). No parallel mechanism.3. Two-phase effect, honestly reported
cancel_requestedwhen the abort is issued.cancelledonly when the worker's settlement is DELIVERED on the settle path with a terminaldown.not_livewhen the worker already settled, or when it settlesdonedespite the abort — its result stands and the operation terminated nothing.unknown. A missing process never reads as success.4. One export mirroring
writeWorkerSteercancelWorker(eventDir, worker, operationId, { reason?, source? })returns the operation's current durable state;readWorkerCancellation(eventDir, operationId)is the pure reader a reconnecting client uses. Both exported fromsrc/runtime/index.tsbeside the steer exports.5. Descendant naming
terminatedset names the requested worker plus every subtree node id with a terminaldown/cancelledjournal record at or afterrequestedAt, read from the durable spawn journal beside the layout. The nested trees journal their terminal records before the lead settles (closeNestedScopedrains them), so the evidence is durable by reconcile time.6. The TUI bypass
cnow goes throughcancelWorker— one writer, one acknowledger, an honest notice showing the acknowledged effect.cis genuinely ROOT-scoped: it cancels the whole run, and a non-retained supervisor tree exposes no root handle to acknowledge with. Per the audit, that is a separate contract gap, so the root-level path stays acancel.request.jsonwrite for the host process to honor, and its notice now says so (host-honored; not runtime-acknowledged) instead of implying acknowledgement. It is not folded into this worker-scoped contract.Wiring:
supervise({ runDir })passes the run directory ascontrolDirto the root manager (supervise.ts,SupervisorAgentDeps.controlDir), so every durable run acknowledges cancels with no extra configuration. The acknowledger resolves references against its OWN workers; a request naming a deeper descendant stays unanswered — cancel its lead instead (documented oncontrolDir).Acceptance tests (
tests/kernel/worker-cancellation.test.ts, all real runtime runs over a file-backed journal)repeating one operationId applies cancellation once and returns the same record(abort listener fired once; one request line; deep-equal record)cancelling one child leaves siblings running until they settle normally(sibling live through the abort, settlesdone, wins the run)cancelling a lead names every terminated descendant while a sibling subtree survives(terminated = [lead, lead:s0, lead:s1]; peer subtree delivers)a reconnecting client reads the acknowledged result from the layout alonea worker that is already gone acknowledges not_live, and an unknown one stays unknown — never successthe cancelled worker reaches a terminal down state visible on the settle path(bussettledevent + durable journal record)Docs
docs/canonical-api.mddecision table gains the cancel row beside the steer row.docs/api/*regenerated withpnpm run docs:api;docs:freshnessreports OK (it reported 21 drift issues before this branch).Verification
pnpm typecheckclean.tests/kernel/worker-cancellation.test.ts: 6/6 pass.pnpm testrun on this branch (results below in first comment / CI).