Tower serves API requests before internal wiring completes — terminal DELETE returns spurious 404 during startup window

[waleedkadous]

Problem

Tower opens its HTTP port and accepts requests before its internal dependency wiring is complete. Requests that need _deps (set by initInstances()) during that window fail closed. Concretely: DELETE /api/terminals/:id returns 404 for a terminal that exists — while GET on the same id returns 200, because the GET path doesn't touch _deps.

Root cause

The entire boot sequence lives inside the server.listen() callback (tower-server.ts:375): stale-socket cleanup → session reconcile → orphan kill → husk sweep (#1227) → session-log retention scan (#1238/#1243) → and only then initInstances() (tower-server.ts:575). killTerminalWithShellper() (tower-instances.ts:799) starts with if (!_deps) return false, which the DELETE route (tower-routes.ts:842) translates into 404 Session not found.

The race predates 3.2.4, but the husk sweep and retention scan added in the 3.2.4 window run between port-open and deps-wiring, so the window now scales with disk state: with ~1,400 session-log files (2.2 GB) it exceeds 100ms, enough that afx tower start && <immediate afx command> can hit it. Discovered when tower-terminals.e2e.test.ts > DELETE > kills and removes a terminal began failing deterministically on a log-heavy dev machine while CI (fresh runners, no logs) stayed green. Instrumentation confirmed _deps was null at DELETE time; a 400ms delay before DELETE makes the test pass.

Suggested fix

1. Routes that need _deps should return 503 with Retry-After ("Tower is starting up") instead of 404/false — generalizing the pattern stopInstance already uses ("Tower is still starting up. Try again shortly.").
2. Structurally: defer server.listen() until after initInstances() (or hold requests until init completes). Note the ordering comments in the boot sequence — some steps must run before initInstances() reads architect state — so this needs care.

(1) is small and safe; (2) closes the window for good if it proves compatible with the ordering constraints.

Testing notes

The existing DELETE e2e test covers the regression once the window is closed (it currently only fails on machines with large session-log volume — CI won't catch it until the fix makes readiness deterministic).

[waleedkadous]

On it! Working on a fix now.

[waleedkadous]

Fixed in PR #1263 (merge commit 757656a).

Completion statistics

Metric	Value
Protocol	BUGFIX
Net diff	447 insertions / 32 deletions across 9 files
Files changed (source)	tower-server.ts, tower-instances.ts, tower-routes.ts
Tests added	2 e2e (tower-startup-readiness.e2e.test.ts") + 1 unit (tower-routes.test.ts`)
CMAP verdicts	gemini=APPROVE, codex=APPROVE, claude=APPROVE (all HIGH, no key issues)
CI	Green on re-run; first run hit an unrelated flake (send-integration.e2e.test.ts afterAll 10s teardown timeout, all 168 tests passing)

What was wrong

The whole boot sequence lived inside the server.listen() callback with initInstances() — which sets _deps — as its last step. killTerminalWithShellper() returns a bare false when _deps is null, and the DELETE route read that as "not found".

Two findings beyond the original report:

1. afx tower start's readiness signal was itself dishonest. It polls /api/status for a 200, which the pre-fix Tower returned during the window because getInstances() returns [] when _deps is null. That is what made afx tower start && <immediate afx command> racy.
2. The window is not exotic. Reproduced deterministically on a machine with an empty log dir, just by connecting in a tight loop and firing the first request at the instant of the bind (the e2e helper's 200ms poll is what usually hides it).

What changed

1. Readiness gate — the port still binds first (it is the single-Tower mutex and what every probe connects to), but requests are held until bootSequence() calls markBootComplete(). A boot exceeding 20s answers 503 + Retry-After rather than hanging clients.
2. Boot reorder — initInstances()/initCron() moved ahead of the husk sweep (Codev process fleet drives macOS memory-pressure kills: stranded shellper husks accumulate per restart + no idle-fleet memory policy #1227), the session-log retention scan (agent-farm: PTY session logs have no rotation or retention policy #1238) and initTunnel(), which now run post-readiness. Readiness should depend on wiring, not housekeeping; and gating it on a remote tunnel connect would make an unreachable cloud endpoint look like a broken local Tower.
3. Second line of defence — instancesReady() lets the _deps-dependent delete paths answer 503 instead of 404, or a lying 204 on the workspace tab-delete path that discards the kill result.

arch.md's canonical boot-order table was stale as of this change; it is updated, with a new "Boot Readiness Gate (#1261)" section stating the invariant for future startup work: anything a request handler depends on must be wired before markBootComplete(), anything else goes after it.

Known adjacent case, deliberately left open

WebSocket upgrades bypass the gate (setupUpgradeHandler attaches its own 'upgrade' listener). Not reachable in practice — every client makes an HTTP call before opening a socket — and the #997 whenStartupReconcileSettled() barrier already covers the WS terminal-upgrade routes for reconcile's output. Worth a follow-up issue if you want it closed explicitly.