Skip to content

fix(devserver): Reap daemons orphaned by an unclean shutdown - #120774

Draft
armcknight wants to merge 1 commit into
masterfrom
andrewmcknight/devserver-reap-orphaned-daemons
Draft

fix(devserver): Reap daemons orphaned by an unclean shutdown#120774
armcknight wants to merge 1 commit into
masterfrom
andrewmcknight/devserver-reap-orphaned-daemons

Conversation

@armcknight

Copy link
Copy Markdown
Member

Starting a devserver now terminates daemons that a previous devserver leaked, so they no longer pile up across runs.

Honcho terminates its children on a clean exit, but when the devserver dies to SIGKILL, a hard terminal close, or a crash they reparent to init and keep running. Nothing noticed them on the next startup, so every unclean shutdown left another set behind.

Leaked consumers are by far the worst of these. They never rejoin their consumer group, so they respawn a multiprocessing worker every few seconds — each paying a full Django import — and hammer the Kafka group coordinator while doing it. On one laptop 16 leaked getsentry-outcomes consumers had accumulated over a week of normal work. With no devserver running at all, they held an idle Kafka container at 185% CPU and burned about 2.7 cores in total; killing them dropped Kafka to 1.3% and total container CPU from ~231% to ~27%.

Before daemons are handed to honcho, a process is reaped only when all of these hold:

  • it is reparented to init, so a daemon owned by a running devserver is never a candidate;
  • its argv matches a daemon this invocation is about to start, as an exact trailing match with the executable basename immediately before it;
  • its working directory is this checkout's, so a devserver in another checkout is untouched;
  • it belongs to the current user.

Worth a look during review: the ppid and working-directory guards are what keep this from touching a live devserver or a second checkout, and _daemon_matches is deliberately strict about the argv tail so that run consumer ingest-events cannot be mistaken for run consumer getsentry-outcomes.

Two details that are easy to get wrong here, both verified against real processes rather than assumed:

  • SIGKILL escalation is load-bearing, not defensive. Wedged consumers never complete their SIGTERM shutdown — pkill on all 16 left every one alive and unchanged. Without the escalation this fix would do nothing for exactly the case that motivates it.
  • Matching on the executable path does not work. In a venv, psutil.Process.exe() resolves to the real interpreter (for example a uv-managed CPython), not the venv, so scoping by sys.prefix would have silently matched nothing. Working directory is used instead, which also covers the node-based watchers. The cwd comparison needs realpath on both sides, since psutil reports an already-resolved path.

Reaping is on by default and can be skipped with --no-reap-orphans.

Honcho terminates its children on a clean exit, but when the devserver dies to SIGKILL, a hard terminal close, or a crash they reparent to init and keep running. Nothing noticed them on the next startup, so they accumulated across runs.

Leaked consumers are the worst of these. They spin forever failing to rejoin their consumer group, respawning a multiprocessing worker every few seconds that each pays a full Django import, and they hammer the Kafka group coordinator while doing it. On one machine 16 had piled up over a week, holding an idle broker at 185% CPU and burning roughly 2.7 cores in total with no devserver running.

Before handing daemons to honcho, terminate any process that is reparented to init, matches a daemon this invocation is about to start, and shares its working directory. SIGKILL escalation is required rather than defensive: wedged consumers never complete their SIGTERM shutdown. Opt out with --no-reap-orphans.
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant