test(search): re-judge the query whose judged file was deleted - #104
Merged
Conversation
`why does an answer with nothing in it still suggest another call` named two files. One of them, internal/adapters/inbound/mcp/handler_intent.go, was deleted in a3a29d2 along with the find_by_intent tool, so the answer key counted two files where at most one could still be found. #99 listed it as a known gap and said it needed re-judging rather than a ranker fix. This is that re-judging. The other named file no longer answers it either. handler_query.go stopped building next actions when the hand-off to find_by_intent went; the only `next` it still carries is the paging offset. The behaviour moved rather than disappeared, and the deletion commit is narrower than it first reads. It says "search's empty answer no longer suggests a hand-off", but that is about the hand-off to a second index, which no longer exists. Two branches of wire.nextActions still fire on a page with nothing on it: the one offering include_weak when every candidate was filtered out as weak, and the annotate step when nothing shown justified itself. So the query is answerable, and now names: internal/app/search/wire/wire.go internal/adapters/inbound/mcp/handler_describe.go wire.go holds nextActions and, beside it, NextAction's own comment, which is the "why" the question asks for in as many words — naming the tool and its arguments turns a dead-end count into a step. handler_describe.go is judged with it as the purest form of the question: describe returns actions only when the target was not found, and a found outline gets none, so there the empty answer is the only one that suggests a call. What it costs, stated plainly, because a re-judgment that raises a number is the move the golden set otherwise forbids: ccg ALL 0.720 (121/168) → 0.726 (122/168), MRR 0.636 → 0.638 ccg ANSWERABLE 0.811 (120/148) → 0.818 (121/148), MRR 0.688 → 0.690 top1 and top3 flat. cobra, context-diary and gorm untouched. The +1 is wire.go, which the intent pool already carried at hit 7, file 6 — inside the page of ten. It is worth being exact about why it scores: the hit is wire.Limits, not nextActions or NextAction. `relevant_files` judges at file granularity, so any hit in a judged file counts it found, and by that rule this is a legitimate hit. But the declaration that matched is not the declaration that answers the question, and a reader landing there still has to read on. The entry keeps file granularity because that is the shape it and its bucket already had; changing the granularity of one query is a different decision from correcting its answer key, and does not belong in the same change. handler_describe.go is at hit 30, file 22, and handler_query.go — no longer judged — is at hit 24, file 21. Both are far past the page. The zeroScoreNotes entry is gone, because the guard added in #97 refused to let a scoring query stay filed as a zero: "why does an answer with nothing in it still suggest another call" no longer scores zero in the baseline; drop its zeroScoreNotes entry so the list cannot rot into a silent excuse That is the guard working as intended, and it is the reason this ticket could not have been quietly absorbed into another change. No production code moves here. queries.json, baseline.json, the guard's own notes and testdata/README.md are the whole diff. Closes #100 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The commit before this moved the ccg totals and left the block in testdata/README.md at the numbers from before it. That block is `make search-eval`'s own output copied whole, and the paragraph under it says hand-drift is how it went stale once already. Copied from the harness on this base, not typed: ALL 0.744 (125/168) 47 62 0.644, ANSWERABLE 0.838 (124/148) 46 61 0.697.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100. Rebased onto
mainafter #101 and #107 landed; every number below is measured on that base.What this is
A re-judgment of one golden query, with no code beside it. The query
why does an answer with nothing in it still suggest another callnamed two files that no longer answer it:handler_intent.gowas deleted withfind_by_intentina3a29d2.handler_query.gostopped building next actions when that hand-off went; the onlynextit still carries is the paging offset.Re-judged onto the two files that do answer it now:
internal/app/search/wire/wire.go, which is wherenextActionsand theNextActioncontract live, andinternal/adapters/inbound/mcp/handler_describe.go, which is the purest form of the same question —describeNextActionsreturns actions only when the target was not found.Movement
top1 and top3 flat. cobra, context-diary and gorm untouched — no fixture was recaptured.
The +1 is
wire.go, and it is worth being exact about why: it scores throughwire.Limits, not throughnextActionsorNextAction. The judgment is still right — that file is where the behaviour the query asks about lives — but nobody should read the gain as the ranker having found the next-action code.Guard
The query's
zeroScoreNotesentry is gone, because it now scores. The guard is symmetric and fails on a listed entry that starts scoring, so the entry had to go rather than being left as a stale excuse. The header count follows: nineteen entries, five of them the ranker's own.testdata/README.mdrecords that an entry has now left that list by being re-judged rather than by being fixed, and states the rule that allows it — a re-judgment may move a number only when it lands as its own change with no code beside it, and the commit says which file scored and why.Verification
CGO_ENABLED=1 go test -tags "fts5" ./... -count=1— all packages passgo vet ./...— cleangofmt -l .— only the two pre-existing parse fixtures undertestdata/binding_gap/go/7fe94f3:verify,verify-postgres🤖 Generated with Claude Code