[Bugfix #1261] Hold Tower API requests until boot wiring completes

[waleedkadous]

Summary

Tower bound its HTTP port and started answering requests before its internal dependency wiring was complete. DELETE /api/terminals/:id returned 404 for a terminal that existed — while GET on the same id returned 200, because the GET path never touches _deps.

This makes the port-bind and "ready to serve" two different things: the port still binds first, but requests are held until the boot sequence has wired up everything a request handler depends on.

Fixes #1261

Root Cause

The entire boot sequence lived inside the server.listen() callback, and initInstances() — which sets tower-instances.ts's _deps — was its last step. Anything arriving before it got whatever a half-wired Tower could produce:

• killTerminalWithShellper() (tower-instances.ts) opens with if (!_deps) return false.
• The DELETE route reads that bare false as "no such terminal" and answers 404.

Two aggravating factors, both confirmed while investigating:

1. The window scales with disk state. The Codev process fleet drives macOS memory-pressure kills: stranded shellper husks accumulate per restart + no idle-fleet memory policy #1227 husk sweep and the agent-farm: PTY session logs have no rotation or retention policy #1238 session-log retention scan both ran between the bind and initInstances(). On a machine with ~1,400 session logs the window exceeds 100ms; on a fresh CI runner it closes before any test's first request lands. That asymmetry is why tower-terminals.e2e.test.ts > DELETE failed deterministically on a log-heavy dev machine while CI stayed green.
2. afx tower start's readiness signal was dishonest. It polls /api/status and treats a 200 as "Tower is up" (commands/tower.ts), but /api/status answered 200 during the window — getInstances() returns [] when _deps is null. So the CLI reported success mid-window, which is what makes afx tower start && <immediate afx command> racy.

Reproduced deterministically before touching anything, with an empty log dir, by connecting in a tight loop and firing the first request at the instant of the bind:

POST   /api/terminals      -> 201 (269ms after port open)
DELETE /api/terminals/:id  -> 404 {"error":"NOT_FOUND", ...}
GET    /api/terminals/:id  -> 200   <-- the terminal is right there

The window is not exotic. It just needs a client that doesn't sleep before its first request.

Fix

1. Readiness gate (tower-server.ts). The listen callback now only logs and kicks off a named bootSequence(); http.createServer holds each request until that sequence calls markBootComplete(). The port still binds first on purpose — it is the single-Tower mutex (a second afx tower start needs EADDRINUSE) and what every readiness probe connects to. A boot that exceeds BOOT_READY_TIMEOUT_MS (20s) answers 503 + Retry-After rather than hanging clients forever. A throw during boot still exits the process, exactly as before (it was an unhandled rejection when this code was the listen callback).

2. Boot reorder. initInstances() + initCron() move up to just after killOrphanedShellpers(); the two maintenance sweeps and initTunnel() now run after the gate opens. Readiness should depend on wiring, not on housekeeping — and gating it on initTunnel() in particular would make an unreachable cloud endpoint look like a broken local Tower. Both sweeps keep their real ordering constraint (after reconcileTerminalSessions), and neither can touch a session created by an incoming request: the husk predicate requires a shellper aged past a 1h grace, and the log sweep only unlinks logs older than the retention window. Measured effect: ready at ~90ms instead of after the disk scans.

3. Second line of defence. New instancesReady() lets routes tell "not wired yet" apart from "no such thing". The DELETE route answers 503 instead of 404, and the workspace tab-delete path — which discards the kill result and answers 204 either way — refuses instead of reporting a close it never performed. Unreachable while the gate holds, and deliberately so: the failure mode if the gate is ever bypassed should be honest rather than misleading.

Test Plan

New tower-startup-readiness.e2e.test.ts. The race was invisible to the suite for two separate reasons, both addressed:

• The e2e helper polls the port every 200ms, which usually lands well after boot finishes → waitForPortImmediate() / startTower({returnAtBind: true}) return at the instant of the bind.
• The window's width depends on the machine's process table and log volume → AF_TEST_BOOT_DELAY_MS widens it to a known duration, so the test asserts an invariant instead of a timing accident.

Two e2e tests: a request issued at bind time gets 204 (not a spurious 404) and the terminal really is gone afterwards; and /api/status does not answer 200 until wiring completes, which is the signal afx tower start depends on. Plus a tower-routes unit test for the instancesReady() 503 guard.

Verified the tests catch the regression: with the gate disabled in dist, both fail — DELETE 503 (the second-line guard firing) and the status-hold assertion at 2ms against a required ≥1200ms.

Suite	Result
Unit (vitest run)	3751 passed, 48 skipped
E2E (vitest.e2e.config.ts)	168 passed, 18 skipped
tsc --noEmit	clean
porch checks (build + tests)	✓ build, ✓ tests

Notes for review

• WebSocket upgrades are not gated. setupUpgradeHandler attaches its own 'upgrade' listener, so it bypasses the HTTP gate. Not reachable in practice — every client makes an HTTP call before opening a socket, so by then the gate is open — and the tower: reconcile terminal sessions before serving requests (kill the restart successor-lookup race) #997 whenStartupReconcileSettled() barrier already covers the WS terminal-upgrade routes for reconcile's output specifically. Left alone rather than silently widening scope; happy to file a follow-up if you'd rather close it explicitly.
• Relationship to tower: reconcile terminal sessions before serving requests (kill the restart successor-lookup race) #997. That barrier gates one dependency for the readers that name it. This gate covers the whole HTTP surface, so whenStartupReconcileSettled() is now always settled for HTTP requests — but it still does real work for WS upgrades, so nothing was removed.
• arch.md's canonical boot-order table was stale as of this change; it's updated, along with a new "Boot Readiness Gate (Tower serves API requests before internal wiring completes — terminal DELETE returns spurious 404 during startup window #1261)" section stating the invariant for future startup work: anything a request handler depends on must be wired before markBootComplete(), anything else goes after it.