gather: deterministic context-pack tool (aft_gather)

[iceteaSA]

gather: deterministic context-pack tool (aft_gather)

What

One new tool — aft_gather — assembles a bounded "context pack" (ranked, deduped, budgeted verbatim code evidence) in a single call. It replaces the multi-turn search → outline → zoom → callgraph read chain an agent otherwise runs to build context around a question or a symbol.

Two modes (mutually exclusive):

• question: "how does X work?" — seeds from handle_semantic_search (same pipeline as aft_search, all lanes/fallbacks)
• symbol + filePath — seeds from the callgraph (impact depth-1 callers + call_tree depth-1 callees)

Seeds expand one hop through the callgraph, dedupe by canonicalized (file, symbol) with seeds winning, and render via render_symbol_within_budget until a hard line budget (default 400, cap 800) is spent. Everything past the cut appears as one-line stubs under ## Beyond budget (zoom to expand) — nothing is silently dropped.

Why

Agents burn serial turns assembling context: search, then outline the hits, then zoom the symbols, then chase callers. Each turn round-trips through the model. A pack returns evidence (verbatim bodies with file:line headers), not conclusions — the agent reasons over it directly, ready to attach to a subagent dispatch.

Measured on a real config repo (3 questions, one call each vs. the manual chain):

• "how does the lane-verdict cache flow end-to-end" → complete 4-file chain (cache shapes, verdict logic, write/read, plugin deny hook) in one pack, used=283/400. Manual baseline: 5-6 tool calls.
• "how does score-tap decide when to write an executor row" → the full decision chain (event filter → dedup → context sources → verdict parse → DB insert) in one pack at budget=200.
• An unrehearsed cross-file question → correct core symbols plus their docstrings, first try.

Independently reproduced on the aft codebase itself: question: "how does bash output compression dispatch pick a compressor" → seeds=15, used=226/400, one pack assembling the full dispatch chain (compress → gate → compress_with_registry_exit_code 20-compressor array → Compressor trait → install path → subc mirror) that otherwise takes a 4–5-call search→zoom chain.

Honest degradation

The pack never lies about its own quality:

• While the semantic index is building, long NL queries degrade to lexical-only file-level hits. The pack renders them as visible file:line (no containing symbol) stubs and flags the header with degraded=semantic-index-building (partial results — retry when index ready) — detected via the response's semantic_status field, cleared as soon as one real seed resolves. No blocking or retry inside the tool.
• Grep-fallback hits ({file, line_text, line}, no symbol name) resolve to their containing symbol by line containment — definitions, call sites, and comment hits all upgrade to the enclosing symbol. Hits with no containing symbol stay visible as stubs.
• Unresolved external/stdlib callees collapse to one summary line ((N unresolved external calls omitted)) instead of drowning the stub list; unresolved seeds and callers are never suppressed.

Implementation

• crates/aft/src/commands/gather.rs — Rust-side composition: calls handle_semantic_search / impact_result / call_tree_result / render_symbol_within_budget directly (shared &AppContext, no bridge round-trips, no parallel reimplementation of search).
• Wiring: main.rs dispatch arm, subc_translate.rs mapping, TS factory packages/opencode-plugin/src/tools/gather.ts + registration (same tier as aft_callgraph — depends on the callgraph store).
• No new dependencies, no config surface beyond the tool args, no LLM calls, no caching.
• Follows the tri-state honest-reporting convention (protocol.rs Response doc-comment): success:false+code for un-performable calls (e.g. invalid_request on a bad mode combo), success:true with a visible degraded/stub pack for partial results — never a bare empty success.

Tests

22 unit tests in gather.rs, including red-checked regressions (each confirmed to fail against pre-fix code): mid-codepoint truncation panic, duplicate-symbol line-anchored resolution, abs/rel path dedupe, containing-symbol resolution via a real TreeSitterProvider, callee-only stub suppression driven through the production build_pack path, and degradation-flag presence/absence/mixed cases.

Limitations (deliberate scope)

• 1-hop expansion only — multi-hop was deliberately excluded: the budget math and ranking get harder and the packs get noisier.
• Callgraph-dependent by design: neighbor expansion quality follows the callgraph store's freshness, same as aft_callgraph.
• Budget counted in lines, not tokens — matches aft's existing budget idiom across zoom/outline.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds a deterministic context-pack builder: gather in Rust and aft_gather_context in @opencode. It replaces the multi-step search→outline→zoom→callgraph chain with one call that returns ranked, deduped, verbatim code within a fixed line budget.

• Two exclusive modes: question (semantic seeds) or symbol + path (impact callers + call-tree callees); 1-hop expansion; dedupe by (file, symbol) with seeds winning.
• Hard line budget (default 400, max 800) with per-symbol balancing; overflow becomes visible stubs under “Beyond budget”; exact used lines computed post-render.
• Honest degradation: grep-fallback hits resolve to containing symbols; no-symbol hits render as stubs; header flags degraded=semantic-index-building and neighbors=skipped(callgraph-unavailable).
• Optional includeTests to include test-file neighbors (default off).
• Transport and surface: wired into subc manifest/schemas/translate; exposed only on the “all” surface; public path maps to internal filePath (translator accepts both). Schema inventory updated (22 bare tools).
• Path normalization/dedupe: absolute vs. relative paths dedupe on Windows; separators normalize before compare; Unix backslash in filenames normalizes to / and renders a clear not-found stub.
• Tests: 22 unit tests plus transport-invariance and an integration test covering includeTests. No new dependencies.

^{Written for commit 28930e0. Summary will update on new commits.}

Greptile Summary

This PR introduces aft_gather — a single-call "context pack" builder that replaces multi-turn search → outline → zoom → callgraph chains. It seeds from either a natural-language question (via handle_semantic_search) or a symbol+filePath pair (via impact+call_tree), expands one callgraph hop, deduplicates by canonicalized (file, symbol), and renders verbatim symbol bodies within a hard line budget, listing overflow as visible stubs.

• New gather.rs (1665 lines): core pack assembly with budget accounting, degradation flags, path normalization, grep-fallback symbol resolution, and 22 unit tests including red-path regressions. A collect_callgraph_neighbors path passes normalized-relative seed paths to impact_result/call_tree_result while symbol mode uses ctx.validate_path-resolved absolute paths for the same functions — a mismatch that could silently produce empty neighbor sets in question mode.
• subc_translate.rs: translate_gather + gather_file_path helper correctly accepts both "path" (schema-advertised) and "filePath" (internal) spellings, resolving the previously-flagged schema-field bug. Schema-exhaustiveness test added.
• gather.ts + wiring: TypeScript adapter exposes path, renames to filePath before bridge dispatch; registered as ALL_ONLY on the OpenCode surface alongside aft_callgraph.

Confidence Score: 4/5

• Safe to merge with the callgraph path inconsistency investigated or accepted — the failure mode is silent degradation (fewer neighbors), not corruption or incorrect output.
• The core budget accounting, dedup, degradation flagging, and stub rendering are well-covered by 22 unit tests and the transport-invariance test. The schema/path dual-spelling fix (gather_file_path) and the include_tests thread-through are correct. The one open question is whether collect_callgraph_neighbors silently produces empty neighbor sets in question mode: it calls impact_result and call_tree_result with normalized-relative seed.file paths while every other call site in the codebase uses absolute validated paths, and errors are swallowed with if let Ok. If the callgraph store is keyed on absolute paths, question-mode neighbor expansion returns nothing with no visible indicator. The integration test covers only symbol mode, so this path is untested end-to-end.
• crates/aft/src/commands/gather.rs — specifically collect_callgraph_neighbors lines 421–447 and the absence of a question-mode callgraph expansion integration test.

Important Files Changed

Filename	Overview
crates/aft/src/commands/gather.rs	New 1665-line core implementation. collect_callgraph_neighbors passes normalized relative seed.file paths to impact_result/call_tree_result while symbol mode uses absolute validated paths for the same functions; errors are silently swallowed, making question-mode neighbor expansion potentially invisible. Also contains unreachable third dedup loop. Otherwise the budget accounting, degradation flags, path normalization, and suppression logic are well-designed with thorough unit tests.
crates/aft/src/subc_translate.rs	Adds translate_gather with a gather_file_path helper that correctly accepts both "path" (schema-advertised) and "filePath" (internal/OpenCode) spellings. The previously-reported schema-field mismatch bug is resolved. New tests cover both spellings, includeTests forwarding, and schema-key exhaustiveness.
packages/opencode-plugin/src/tools/gather.ts	New TypeScript adapter. Exposes path externally, renames it to filePath before dispatching to the Rust bridge. Mode validation mirrors translate_gather. Transport-invariance test covers the key rename and confirms no internal transport names leak into args.
crates/aft/tests/integration/gather_test.rs	Integration test covers includeTests flag in symbol mode (hiding vs. showing test-file callers) with a real filesystem fixture. No integration test exercises question mode's callgraph neighbor expansion path, which is the path most likely to be affected by the relative-path inconsistency.
crates/aft/src/subc/manifest.rs	Registers gather in the agent-core tool set, HeavyInit lane (same tier as callgraph/semantic_search), and the manifest tool list. Manifest test updated. Changes are consistent with existing patterns.
packages/opencode-plugin/src/tool-registration.ts	Adds aft_gather_context to ALL_ONLY_TOOLS and spreads gatherTools(ctx) unconditionally, consistent with how other ALL_ONLY tools like aft_refactor are registered (surface filtering is handled by normalizeToolMap).

Sequence Diagram

sequenceDiagram
    participant Agent
    participant gather.ts
    participant translate_gather
    participant handle_gather
    participant handle_semantic_search
    participant collect_callgraph_neighbors
    participant build_pack

    Agent->>gather.ts: "aft_gather_context({question|symbol+path})"
    gather.ts->>gather.ts: rename path→filePath, validate mode
    gather.ts->>translate_gather: gather_file_path() accepts "path" or "filePath"
    translate_gather->>handle_gather: "{command:"gather", filePath:...}"

    alt question mode
        handle_gather->>handle_semantic_search: "top_k=15 seeds"
        handle_semantic_search-->>handle_gather: results + semantic_status
        handle_gather->>collect_callgraph_neighbors: seeds (normalized-relative file paths)
        Note over collect_callgraph_neighbors: impact_result(seed_path) where seed_path is relative<br/>symbol mode uses ctx.validate_path (absolute) — inconsistency
        collect_callgraph_neighbors-->>handle_gather: neighbors (or silently empty)
    else symbol mode
        handle_gather->>handle_gather: ctx.validate_path → absolute file_path
        handle_gather->>handle_gather: "impact_result(&file_path) — absolute ✓"
    end

    handle_gather->>build_pack: seeds + neighbors + budget
    build_pack->>build_pack: dedup, render within budget, stub overflow
    build_pack-->>Agent: gather pack text with file:line headers

Loading

_{Reviews (29): Last reviewed commit: "gather: deterministic context-pack tool ..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 7 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[iceteaSA]

Follow-up on the two maintainability notes from the Greptile summary (they weren't separate review threads, so noting here) — both addressed in cf09b9b9:

• degraded expression flagged as always-false in the non-empty-seeds path — intentional: degraded only applies when zero symbol-level seeds resolve (a building index yields no seeds; a ready index with seeds is never degraded). Logic unchanged; added a comment on the why so it doesn't read as dead code.
• callee-suppression string-match coupling — extracted UNRESOLVED_MARKER ("(symbol not resolved)") and CALLEE_PROVENANCE_PREFIX ("callee-of-") consts, referenced by both the producer (render_symbol_section err arm / collect_callees_for_seed) and the consumer (build_pack suppression guard) so the match can't silently drift. Only test-site literals remain, intentionally — a test pointing at the const couldn't catch const drift.

Matched/rendered text is byte-identical; 23/23 gather tests green.

[iceteaSA]

Rebased onto current main (690bed59) and renamed the tool to aft_gather_context — the longer name reads more clearly at the call site.

The rename covers the advertised tool name and the agent-facing error strings only. The wire command stays "gather" (TS→Rust transport), as do the file and function names, matching how aft_callgraph maps to callers/call_tree/impact from navigation.ts. Both manifest entries were updated: the REG-V049-OC-ALL set and the HOSTONLY-V049-006 allowlist row.

Rebase conflicts were confined to ARCHITECTURE.md and STRUCTURE.md; all code auto-merged. tools/structure.ts and tools/lsp.ts are gone from main, so I kept your shorter tool lists and only re-inserted gather.ts.

Verified: registration-parity 10, tool-surface-transport-invariant 2, tools 10, cargo test gather 23, bun run build and cargo build clean. The built bundle contains aft_gather_context and zero occurrences of the old name.

Still one commit. Two notes on the full plugin suite, both pre-existing on main rather than from this branch: the two e2e outline command failures and the format_on_edit ones. main's own CI is currently red on 1d1a6968, 1da22bfe, and ef57355f, and 690bed59's run was cancelled — happy to hold this until that's sorted if you'd rather rebase onto a green base.

[iceteaSA]

Opened #192 to ask the design question separately from the diff: whether you want one-call context packs in aft at all, and in what shape. Answer there and I'll rework this to match — or close it if the answer is no.

Current state here: green on a4b6aaee, 14/14 including the Windows jobs, still one commit.

[iceteaSA]

Rebased onto v0.49.3 (0dbcb541). Clean rebase, still one commit, now 500f1a9a. Regenerated the four governed manifests for the new base.

CI: 13 green, one red — E2E / E2E (Windows native). I believe it's an infra flake, not the diff, and here's the evidence rather than an assertion.

Scenario 1's opencode run timed out at 90s and the process was stopped. The whole S1 failure set follows from that one timeout:

(opencode run timed out at 90s -- stopping process)
PASS [plugin loaded]
FAIL [bridge spawned]
WARN [search index started (non-blocking)]
FAIL [aimock received chat-completion requests]
Results: 23 passed, 2 failed

plugin loaded passes (the plugin resolved and started) but bridge spawned fails because started, pid never reaches the log — the run was killed before the bridge finished spawning. aimock received chat-completion requests fails for the same reason: opencode never got far enough to talk to the mock. The journal confirms it — 13 requests total, all with S2 timestamps, none from S1.

Same job, same assertions, on the two nearest green runs:

• a4b6aaee (this branch, previous sha): PASS [bridge spawned], PASS [aimock received…], 26 passed / 0 failed, no timeout line
• 0dbcb541 (your main, the base I just rebased onto): identical, 26 / 0, no timeout line

So the same code path passes on the parent commit and on the previous revision of this branch. The delta between them is a 90s wall clock on a cold Windows runner, not the diff — nothing in this change is reachable from bridge spawn or plugin boot. Unit / Cargo test (Windows), Cargo check deny-warnings (Windows), and Bash permission e2e (Windows) are all green on this sha.

I can't re-run it — gh run rerun returns Must have admin rights to Repository. Could you kick that one job? If you'd rather I force-push an empty amend to retrigger the whole suite, say so and I will, but that burns a full round on all 14 jobs to retry one.

Separately: #192 asks whether you want this capability at all, which is the more useful question than another rebase.

[iceteaSA]

Rebased onto 2a2b4f2d and green — ignore the re-run request above, the Windows E2E failure was the flake I described and it cleared on its own.

9bb5af30, still one commit. 14/14 checks pass, E2E / E2E (Windows native) included: PASS [bridge spawned], PASS [aimock received chat-completion requests], 26 passed / 0 failed, no 90s timeout line. That's the same job that failed on the previous sha with the same code in it, which settles the question of whether the diff was involved.

The four new commits didn't touch any file this branch touches, so the rebase was clean and only the governed manifests needed regenerating for the new base.

[iceteaSA]

Green on 4707e578, 11/11 jobs. Rebased onto 10e9119c, still one commit.

Since the last update this branch picked up includeTests (default false) plus three fixes, one of which was a real bug caught in review:

The flag was inert on one path. includeTests threads through six call sites across gather's two modes, and collect_callgraph_neighbors was still passing a literal false to impact_result for callers while forwarding the flag to callees. Question mode would have honoured it for callees and silently ignored it for callers. It survived my own review because false IS the default, so the diff reads correct at a glance. Both paths forward it now, and there is a doc-comment on the function stating the invariant at the point where the next partial conversion would happen.

The coverage gap that allowed it is closed. crates/aft/tests/integration/gather_test.rs drives symbol mode against a fixture with 20 production callers and 5 in src/__tests__/, asserting on gather's actual pack text: no __tests__ by default, __tests__ present with includeTests: true. I proved it non-vacuous rather than trusting a pass — reverting the fixed call site to false yields:

includeTests gather should show test callers; response:
  "## gather pack | mode=impact(...) | seeds=1 | neighbors=15 | budget=400 used=81"
  ...15 production callers, zero __tests__ paths

Restored, it passes with 20 neighbors including the test callers. Red → green → red, all three outcomes observed.

A Windows-only test bug. gather_resolves_relative_file_path_from_project_root hardcoded "/project/src/foo.rs"; on Windows join produces \project\src\foo.rs, so the assertion failed while the resolution was correct. It now derives the expectation from resolve_path_from_project_root. General rule for this repo: a test asserting a path shape has to build that shape the platform's way, or it asserts the developer's OS.

Two red rounds along the way were runner flakes, both cleared on the next sha with no code change touching them — fs_lock (a 250ms timing assertion, Windows) and lsp_diagnostics (publishDiagnostics timeout, macOS). Neither file is in this diff and both pass locally; upstream's own CI was green on the same base each time.

Full gate locally: 1643 tests run: 1642 passed, 1 failed (one environmental sandbox failure from this box's PATH), plugin suites at their known baseline, governed manifests regenerated, release artifacts untouched.

Still the open question from #192, and it is yours rather than mine: whether one-call context packs belong in aft at all. There is now a second person asking for them in that thread with an independently-arrived-at use case.

[iceteaSA]

Greptile's P1 was correct and I've fixed it in 631a4d80. Verified against the built binary before and after, since this is exactly the class of bug that passes a unit suite:

# before — schema-advertised key
{"name":"gather","arguments":{"symbol":"handle_gather","path":"…/gather.rs"}}
→ invalid_request: 'symbol' and 'filePath' must be provided together

# after
→ not_configured: gather: project not configured — send 'configure' first

not_configured is the correct next error for an unconfigured probe process — it means translation succeeded and the call reached the handler.

The schema advertises path (matching zoom and grep), while the OpenCode adapter renames path → filePath before calling the bridge. translate_gather read only filePath, at both the presence check and the copy-to-output step. So first-party traffic worked and every schema-conformant MCP call failed the together-check — regardless of input correctness, since neither site could see the argument.

Fixed by reading either spelling and writing the internal one:

fn gather_file_path(map_in: &Map<String, Value>) -> Option<&str> {
    ["path", "filePath"]
        .into_iter()
        .find_map(|key| map_in.get(key).and_then(Value::as_str))
        .filter(|value| !value.is_empty())
}

On the tests, since the masking is the more interesting part of this report. Both existing tests passed filePath directly, so they exercised the adapter's spelling rather than the one the schema documents — green suite, broken contract. Rather than only flipping them, I added a test that derives the key set from the embedded schema itself:

gather_translator_reads_every_schema_advertised_key reads subc_tool_schemas.json, iterates every advertised property, and asserts the translator accepts each one. Adding a parameter to the schema without wiring it into translate_gather now fails there instead of in production. It also panics on an unrecognised key, so a rename can't silently drop coverage.

Red control on both new tests: removing "path" from the alias list gives schema-advertised 'path' must be accepted: TranslateError { code: "invalid_request", message: "'symbol' and 'filePath' must be provided together" } — the exact production symptom.

Verification: 4222 Rust tests, 1 pre-existing sandbox failure that also fails on clean main in this checkout · lint, format, governed audit clean · CI was 11/11 green on the parent sha 9e147b98.

Also rebased onto c2b016a9 — the branch had gone unmergeable against current main.

[ualtinok]

Closing with #192 — see the rationale there. Thanks for building the exploration out; the neighbor-expansion thinking fed into the read-range hints that did ship.