Put Binary Ninja ahead of Ghidra in the fallback order - #230
Open
subwire wants to merge 2 commits into
Open
Conversation
`export` was added in #222 to replace the load -> list_functions -> decompile -> decompile -> ... loop that everyone otherwise scripts by hand. Across 60 solve sessions on a 40-binary benchmark it was invoked **zero times**. Not because agents rejected it. SKILL.md -- 589 lines, the doc agents actually read -- never mentions `export`. It says "Always start with `list_functions` and `list_strings`", which is precisely the loop export exists to collapse, so agents did that instead. The four sessions that found the verb at all found it by running `decompiler export --help` inside a batch of `--help` calls at command 60-85 of ~130, long after they had done the work by hand. One of them then hand-wrote eight per-function .c files -- a worse copy of export's own output -- having spent 112 CLI calls to get there. A 117-function binary exports in 31 seconds. Two changes: **SKILL.md** now leads the "first moves" section with export, shows the export-then-grep pattern, and says when *not* to use it (very large binaries, or when you already know the one function you want). The interrogative path is still documented, just no longer the only thing documented. **`--min-size`** makes the output worth reading rather than only grepping. Import stubs dominate the function count of any dynamically linked binary: on the crackme above, 93 of 117 functions were PLT thunks and they were 45% of the exported pseudocode. `--min-size 16` cuts that binary to 24 functions and 108K (from 117 and 480K) while keeping every function a human analyst annotated by hand. Opt-in; default 0 exports everything exactly as before. The manifest reports `skipped_below_min_size` so the trim is never silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SKILL.md told agents "use `binja` only when explicitly requested" and documented the fallback chain as `ida → ghidra → angr`, leaving the one other *licensed* backend out of it entirely. The result, measured over 60 solve sessions: binja was loaded in **1**. Agents reached for IDA first every time (20/20 and 38/38 where a backend was named -- the IDA guidance works), and when IDA died they went straight to Ghidra, because that is what the doc says to do. So a paid Binary Ninja license sat unused while the fallback path ran on the unlicensed backend. That fallback is not free. In the same run a Ghidra load died outright on a binary named `timo#3` (see the name-sanitizing fix), costing a backend switch and several minutes on a binary binja would have opened. Fallback order is now `ida → binja → ghidra → angr`, with a pointer to `decompiler backend status binja --json` so an agent can tell in one call whether it is licensed here rather than assuming. Where binja is not installed the guidance is unchanged in effect: the check reports unavailable and the chain falls through to ghidra. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #227 — branched from
export-discoverabilitybecause both editSKILL.md. Review the top commit only; retarget tomainonce #227 lands.The doc routes agents away from a licensed backend
SKILL.mdsaid:So the fallback chain omits the one other licensed backend, and explicitly tells agents not to reach for it.
Measured
Over 60 solve sessions on a 40-binary corpus:
The IDA guidance works — agents reach for it every time. But when IDA dies they go straight to Ghidra, because that is what the doc says. A paid Binary Ninja license sat essentially unused while the fallback ran on the unlicensed backend.
Binja is available in our image, so this was purely the doc:
That fallback isn't free
In the same run, a Ghidra load died outright on a binary named
timo#3(IllegalArgumentException: name contains invalid character, fixed separately in #229). The agent burned a backend switch and several minutes on a binary Binary Ninja would have opened.Change
Fallback order becomes
ida → binja → ghidra → angr, with a pointer todecompiler backend status binja --jsonso an agent can check in one call whether it is licensed here rather than assuming either way.This is deliberately a check-then-use, not an assumption: where binja is not installed, the status call reports unavailable and the chain falls through to ghidra exactly as before.
🤖 Generated with Claude Code