Skip to content

Keep a dead server explainable after its record is pruned - #228

Open
subwire wants to merge 2 commits into
mainfrom
tombstones-on-223
Open

Keep a dead server explainable after its record is pruned#228
subwire wants to merge 2 commits into
mainfrom
tombstones-on-223

Conversation

@subwire

@subwire subwire commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #223 — branched from server-startup-reliability, so review the top commit only. Retarget to main once #223 lands.

The gap

#223 made a dead server say so, instead of reading as "you never started one". It works — for exactly one command.

That command prunes the registry entry. Every command after it goes blind again. And the command that usually gets there first is list, the single most-used verb, which prints a bare:

No running decompiler servers.  (registry: /home/ctf/.local/state/declib/servers)

…which reads precisely like nothing was ever started. The diagnosis is destroyed by the command used to check state.

Measured

On a 40-binary benchmark run, of 79 "missing server" errors:

count
carried the useful diagnosis 10
corpse already eaten by an intervening list 69

Reproduced deterministically

# no `list` in between — works
load → kill -9 → list_functions --id X
  "server X is no longer running (binary: /tmp/target). It died or was
   stopped; its analysis is gone.  Reload it with: decompiler load ..."

# `list` in between — which is what agents actually do
load → kill -9 → list → list_functions --id X
  "No running decompiler server matches {'id': 'X'}."

Change

Pruning leaves a tombstone beside the registry entry, so the death stays explainable for any later command.

  • list now reports recent deaths with the reload command, instead of implying nothing ever ran
  • a successful load of the same binary clears the tombstone, so a stale death is never reported as current
  • tombstones are a hint, not a log: capped at the 16 most recent, and excluded from the live-record glob so one can never be mistaken for a running server

After:

No running decompiler servers.  (registry: ...)

Recently died (analysis is gone; reload to continue):
  095d5185f3   /tmp/target
      reload: decompiler load /tmp/target --backend ida

Tests

Five: pruning leaves a tombstone; the diagnosis survives a second lookup (the actual regression); tombstones are never mistaken for live records; reload clears them; the count is capped.

🤖 Generated with Claude Code

subwire and others added 2 commits July 26, 2026 17:35
Three failures accounted for 54 of the 62 DecLib errors we logged across
a CTF where DecLib was the sole decompiler interface. None of them told
the user what had actually happened, and each one ended with an agent
abandoning the CLI for a hand-rolled script.

**"Failed to open database <binary>" (14x).** Reproduced: it is what you
get when a second server opens a project another server already holds.
`--force` is documented as "run a second copy", but the default project
dir is derived from binary+backend alone, so the second server always
collided with the first and died. Forced copies now get their own project
dir, and `--force` works:

    before: Failed to open database /tmp/orig
    after:  two live servers, project_dir orig-5a3e46a4
                                      and orig-5a3e46a4-722556f738

The bare message also reads like file corruption, so any log carrying it
now gets an explanation naming the real cause and the three ways out
(`--id`, `--replace`, `--project-dir`).

**"Timed out waiting Ns for server to start" (19x).** Two problems. The
advice said "Check backend dependencies (e.g. GHIDRA_INSTALL_DIR)"
whatever the backend was -- actively misleading on IDA. It is now
backend-aware. And the wait was silent, so a legitimately slow load was
indistinguishable from a hang; agents retried at 180s, 240s, then 300s,
burning ~12 minutes before giving up. The wait now reports every 10s,
surfacing the backend's own last log line:

    [10s] INFO | declib.api.decompiler_server | Using headless interface utilizing ida
    [20s] still starting (5s left)...
    Timed out waiting 25s ... Check the IDA install and that its licence/EULA
    is accepted (`decompiler backend status ida`).

**"No running decompiler server matches {...}" (21x).** The registry
prunes dead records during lookup, so a server that died left no trace
and the error read as "you never started one". list_servers/find_servers
can now report what they reaped, and the error names the corpse:

    Decompiler server 078c974380 is no longer running (binary: /tmp/orig).
    It died or was stopped; its analysis is gone.
    Reload it with:
      decompiler load /tmp/orig --backend ida

Deliberately not doing silent auto-revive: re-analysis can take minutes,
and quietly spending them (or quietly losing unsaved annotations) is a
worse surprise than an accurate error naming the command to run.

Tests: backend-aware hints for every backend, the lock explanation fires
only on lock failures, and the registry reports pruned records.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
one". It only works for the *first* command after the death, though: that
command prunes the registry entry, and every command after it goes blind
again.

The command that usually gets there first is `list` -- the single
most-used verb -- and it prints a bare "No running decompiler servers",
which reads exactly like nothing was ever started. So in practice the
diagnosis is destroyed by the command used to check state.

Measured on a 40-binary run: of 79 "missing server" errors, only **10**
carried the useful message. The other **69** had already had the corpse
eaten by an intervening `list`.

Reproduced deterministically:

    load -> kill -9 -> list_functions --id X
        "server X is no longer running ... reload with: decompiler load ..."
    load -> kill -9 -> list -> list_functions --id X
        "No running decompiler server matches {'id': 'X'}."

Pruning now leaves a tombstone next to the registry entry, so the death
stays explainable for any later command. `list` reports recent deaths
with the reload command instead of implying nothing ever ran, and a
successful `load` of the same binary clears the tombstone so a stale
death is never reported as current.

Tombstones are a hint, not a log: capped at the 16 most recent, and
excluded from the live-record glob so one can never be mistaken for a
running server.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant