Keep a dead server explainable after its record is pruned - #228
Open
subwire wants to merge 2 commits into
Open
Conversation
Three failures accounted for 54 of the 62 DecLib errors we logged across
a CTF where DecLib was the sole decompiler interface. None of them told
the user what had actually happened, and each one ended with an agent
abandoning the CLI for a hand-rolled script.
**"Failed to open database <binary>" (14x).** Reproduced: it is what you
get when a second server opens a project another server already holds.
`--force` is documented as "run a second copy", but the default project
dir is derived from binary+backend alone, so the second server always
collided with the first and died. Forced copies now get their own project
dir, and `--force` works:
before: Failed to open database /tmp/orig
after: two live servers, project_dir orig-5a3e46a4
and orig-5a3e46a4-722556f738
The bare message also reads like file corruption, so any log carrying it
now gets an explanation naming the real cause and the three ways out
(`--id`, `--replace`, `--project-dir`).
**"Timed out waiting Ns for server to start" (19x).** Two problems. The
advice said "Check backend dependencies (e.g. GHIDRA_INSTALL_DIR)"
whatever the backend was -- actively misleading on IDA. It is now
backend-aware. And the wait was silent, so a legitimately slow load was
indistinguishable from a hang; agents retried at 180s, 240s, then 300s,
burning ~12 minutes before giving up. The wait now reports every 10s,
surfacing the backend's own last log line:
[10s] INFO | declib.api.decompiler_server | Using headless interface utilizing ida
[20s] still starting (5s left)...
Timed out waiting 25s ... Check the IDA install and that its licence/EULA
is accepted (`decompiler backend status ida`).
**"No running decompiler server matches {...}" (21x).** The registry
prunes dead records during lookup, so a server that died left no trace
and the error read as "you never started one". list_servers/find_servers
can now report what they reaped, and the error names the corpse:
Decompiler server 078c974380 is no longer running (binary: /tmp/orig).
It died or was stopped; its analysis is gone.
Reload it with:
decompiler load /tmp/orig --backend ida
Deliberately not doing silent auto-revive: re-analysis can take minutes,
and quietly spending them (or quietly losing unsaved annotations) is a
worse surprise than an accurate error naming the command to run.
Tests: backend-aware hints for every backend, the lock explanation fires
only on lock failures, and the registry reports pruned records.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
one". It only works for the *first* command after the death, though: that
command prunes the registry entry, and every command after it goes blind
again.
The command that usually gets there first is `list` -- the single
most-used verb -- and it prints a bare "No running decompiler servers",
which reads exactly like nothing was ever started. So in practice the
diagnosis is destroyed by the command used to check state.
Measured on a 40-binary run: of 79 "missing server" errors, only **10**
carried the useful message. The other **69** had already had the corpse
eaten by an intervening `list`.
Reproduced deterministically:
load -> kill -9 -> list_functions --id X
"server X is no longer running ... reload with: decompiler load ..."
load -> kill -9 -> list -> list_functions --id X
"No running decompiler server matches {'id': 'X'}."
Pruning now leaves a tombstone next to the registry entry, so the death
stays explainable for any later command. `list` reports recent deaths
with the reload command instead of implying nothing ever ran, and a
successful `load` of the same binary clears the tombstone so a stale
death is never reported as current.
Tombstones are a hint, not a log: capped at the 16 most recent, and
excluded from the live-record glob so one can never be mistaken for a
running server.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #223 — branched from
server-startup-reliability, so review the top commit only. Retarget tomainonce #223 lands.The gap
#223 made a dead server say so, instead of reading as "you never started one". It works — for exactly one command.
That command prunes the registry entry. Every command after it goes blind again. And the command that usually gets there first is
list, the single most-used verb, which prints a bare:…which reads precisely like nothing was ever started. The diagnosis is destroyed by the command used to check state.
Measured
On a 40-binary benchmark run, of 79 "missing server" errors:
listReproduced deterministically
Change
Pruning leaves a tombstone beside the registry entry, so the death stays explainable for any later command.
listnow reports recent deaths with the reload command, instead of implying nothing ever ranloadof the same binary clears the tombstone, so a stale death is never reported as currentAfter:
Tests
Five: pruning leaves a tombstone; the diagnosis survives a second lookup (the actual regression); tombstones are never mistaken for live records; reload clears them; the count is capped.
🤖 Generated with Claude Code