fix(devserver): Reap daemons orphaned by an unclean shutdown - #120774
Draft
armcknight wants to merge 1 commit into
Draft
fix(devserver): Reap daemons orphaned by an unclean shutdown#120774armcknight wants to merge 1 commit into
armcknight wants to merge 1 commit into
Conversation
Honcho terminates its children on a clean exit, but when the devserver dies to SIGKILL, a hard terminal close, or a crash they reparent to init and keep running. Nothing noticed them on the next startup, so they accumulated across runs. Leaked consumers are the worst of these. They spin forever failing to rejoin their consumer group, respawning a multiprocessing worker every few seconds that each pays a full Django import, and they hammer the Kafka group coordinator while doing it. On one machine 16 had piled up over a week, holding an idle broker at 185% CPU and burning roughly 2.7 cores in total with no devserver running. Before handing daemons to honcho, terminate any process that is reparented to init, matches a daemon this invocation is about to start, and shares its working directory. SIGKILL escalation is required rather than defensive: wedged consumers never complete their SIGTERM shutdown. Opt out with --no-reap-orphans.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starting a devserver now terminates daemons that a previous devserver leaked, so they no longer pile up across runs.
Honcho terminates its children on a clean exit, but when the devserver dies to SIGKILL, a hard terminal close, or a crash they reparent to init and keep running. Nothing noticed them on the next startup, so every unclean shutdown left another set behind.
Leaked consumers are by far the worst of these. They never rejoin their consumer group, so they respawn a
multiprocessingworker every few seconds — each paying a full Django import — and hammer the Kafka group coordinator while doing it. On one laptop 16 leakedgetsentry-outcomesconsumers had accumulated over a week of normal work. With no devserver running at all, they held an idle Kafka container at 185% CPU and burned about 2.7 cores in total; killing them dropped Kafka to 1.3% and total container CPU from ~231% to ~27%.Before daemons are handed to honcho, a process is reaped only when all of these hold:
Worth a look during review: the ppid and working-directory guards are what keep this from touching a live devserver or a second checkout, and
_daemon_matchesis deliberately strict about the argv tail so thatrun consumer ingest-eventscannot be mistaken forrun consumer getsentry-outcomes.Two details that are easy to get wrong here, both verified against real processes rather than assumed:
pkillon all 16 left every one alive and unchanged. Without the escalation this fix would do nothing for exactly the case that motivates it.psutil.Process.exe()resolves to the real interpreter (for example a uv-managed CPython), not the venv, so scoping bysys.prefixwould have silently matched nothing. Working directory is used instead, which also covers the node-based watchers. The cwd comparison needsrealpathon both sides, since psutil reports an already-resolved path.Reaping is on by default and can be skipped with
--no-reap-orphans.