Make server startup and teardown failures diagnosable - #223
Open
subwire wants to merge 1 commit into
Open
Conversation
Three failures accounted for 54 of the 62 DecLib errors we logged across
a CTF where DecLib was the sole decompiler interface. None of them told
the user what had actually happened, and each one ended with an agent
abandoning the CLI for a hand-rolled script.
**"Failed to open database <binary>" (14x).** Reproduced: it is what you
get when a second server opens a project another server already holds.
`--force` is documented as "run a second copy", but the default project
dir is derived from binary+backend alone, so the second server always
collided with the first and died. Forced copies now get their own project
dir, and `--force` works:
before: Failed to open database /tmp/orig
after: two live servers, project_dir orig-5a3e46a4
and orig-5a3e46a4-722556f738
The bare message also reads like file corruption, so any log carrying it
now gets an explanation naming the real cause and the three ways out
(`--id`, `--replace`, `--project-dir`).
**"Timed out waiting Ns for server to start" (19x).** Two problems. The
advice said "Check backend dependencies (e.g. GHIDRA_INSTALL_DIR)"
whatever the backend was -- actively misleading on IDA. It is now
backend-aware. And the wait was silent, so a legitimately slow load was
indistinguishable from a hang; agents retried at 180s, 240s, then 300s,
burning ~12 minutes before giving up. The wait now reports every 10s,
surfacing the backend's own last log line:
[10s] INFO | declib.api.decompiler_server | Using headless interface utilizing ida
[20s] still starting (5s left)...
Timed out waiting 25s ... Check the IDA install and that its licence/EULA
is accepted (`decompiler backend status ida`).
**"No running decompiler server matches {...}" (21x).** The registry
prunes dead records during lookup, so a server that died left no trace
and the error read as "you never started one". list_servers/find_servers
can now report what they reaped, and the error names the corpse:
Decompiler server 078c974380 is no longer running (binary: /tmp/orig).
It died or was stopped; its analysis is gone.
Reload it with:
decompiler load /tmp/orig --backend ida
Deliberately not doing silent auto-revive: re-analysis can take minutes,
and quietly spending them (or quietly losing unsaved annotations) is a
worse surprise than an accurate error naming the command to run.
Tests: backend-aware hints for every backend, the lock explanation fires
only on lock failures, and the registry reports pruned records.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
subwire
force-pushed
the
server-startup-reliability
branch
from
July 26, 2026 17:35
9ef3493 to
094a72d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
We ran DecLib as the sole decompiler interface for a fleet of autonomous CTF agents at D3CTF 2026 and replayed every session afterwards. 54 of the 62 DecLib errors we logged were these three. None told the user what had actually happened, and each one ended with an agent abandoning the CLI for a hand-rolled IDAPython script.
1.
Failed to open database <binary>(14×)Reproduced, and the cause is not what the message suggests. It is what you get when a second server opens a project database another server already holds:
--forceis documented as "run a second copy", but the default project dir is derived from binary+backend alone — so the second server always collided with the first and died. Forced copies now get their own project dir:The bare message also reads like file corruption, so any server log carrying it now gets an explanation naming the real cause and the three ways out (
--id,--replace,--project-dir).2.
Timed out waiting Ns for server to start(19×)Two problems.
The advice read "Check backend dependencies (e.g. GHIDRA_INSTALL_DIR)" regardless of backend — actively misleading when the backend is IDA. It is now backend-aware.
And the wait was silent, so a legitimately slow load was indistinguishable from a hang. Agents retried at 180s, then 240s, then 300s — ~12 minutes of solve budget per abandonment. The wait now reports every 10s, surfacing the backend's own last log line:
(That run is a real 137 MB
libLLVM.so— genuinely slow rather than stuck, which is now visible.)3.
No running decompiler server matches {...}(21×)The registry prunes dead records during the lookup, so a server that died left no trace and the error read as "you never started one".
list_servers/find_serverscan now report what they reaped, and the error names the corpse:Deliberately not doing silent auto-revive. Re-analysis can take minutes, and quietly spending them — or quietly discarding unsaved annotations — is a worse surprise than an accurate error naming the exact command to run. Happy to add it behind an explicit flag if you'd prefer.
Tests
Backend-aware hints for every backend (asserting
GHIDRA_INSTALL_DIRappears for ghidra and only ghidra); the lock explanation fires on lock failures and not on unrelated ones; the registry reports pruned records with enough detail to name the binary.🤖 Generated with Claude Code