test: judge the package nodes annot and crossref already rank first - #105
Merged
Conversation
Two queries name a package by the name that package declares and were judged on everything but the package node itself. `annot` is a prefix query whose key already accepts a prefix match on a name — `graph.Annotation` is judged though the query stops five letters short — and `internal/domain/annotation` declares `package annotation`. `crossref` names a whole feature in one word, and `internal/app/crossref` declares `package crossref`; `impact` was judged on that same reasoning already. `mcp` gains no node. Its `why` now records why the second package node in its pool stays unjudged: `internal/runtime/mcp` declares `package mcpruntime`, and only a directory in its path is spelled `mcp`. This is a re-judgment, so it moves numbers with no code beside it. Per query, measured by rerunning the report with each edit alone: annot relevant 4 -> 5, found 2 -> 3, first relevant rank 2 -> 1 crossref relevant 3 -> 4, found 3 -> 4, rank 1 unchanged mcp unchanged Only the prefix bucket moves: 0.882 (15/17) top1 4 MRR 0.900 becomes 0.895 (17/19) top1 5 MRR 1.000. ccg ALL 0.744 (125/168) 47 62 0.644 becomes 0.747 (127/170) 48 62 0.649; ANSWERABLE 0.838 (124/148) 46 61 0.697 becomes 0.840 (126/150) 47 61 0.703. cobra, context-diary and gorm are untouched. The top1 that annot regains is the one it lost when the package nodes were added to the pool: the annotation package node is what the ranker had already put at rank 1 while the key called it wrong. Nothing about the ranking changed here — only which nodes the key admits are right. testdata/README.md states the rule the three judgments follow, including its two negative cases: a path segment spelled like the query is not the package's name, and containing an already-judged file is corroboration after the name matched, never a reason on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #102. Rebased onto
mainafter #101, #107 and #104 landed; the numbers below are remeasured on that base.What this is
A re-judgment of the golden set, with no code beside it. Two queries name a package by the name that package declares and were judged on everything but the package node itself.
annot— prefix query. Its key already accepts a prefix match on a name (graph.Annotationis judged though the query stops five letters short), andinternal/domain/annotationdeclarespackage annotation.crossref— one word naming a whole feature.internal/app/crossrefdeclarespackage crossref, andimpactwas judged on that same reasoning already.mcpgains no node. Itswhynow records why the other package node in its pool stays unjudged:internal/runtime/mcpdeclarespackage mcpruntime, and only a directory in its path is spelledmcp.A sweep of all four corpora found no other unjudged package node in any top-15 pool: cobra and context-diary have none in any pool at all, and gorm's
clauseis already judged.Movement
Measured by rerunning the report with each edit alone:
Only the
prefixbucket moves:0.882 (15/17) top1 4 MRR 0.900becomes0.895 (17/19) top1 5 MRR 1.000— the same movement as before the rebase; that bucket is untouched by the three PRs that landed in between.cobra, context-diary and gorm are untouched; no fixture was recaptured.
testdata/README.md's## The two totalsblock is updated to the new output, so it does not go stale again.The top1
annotregains is the one it lost in #101 when package nodes entered the pool. The annotation package node is what the ranker had already put at rank 1 while the key called it wrong. Nothing about the ranking changed here — only which nodes the key admits are right.Rule
testdata/README.mdgains the rule these three judgments follow, with its two negative cases: a path segment spelled like the query is not the package's name, and containing an already-judged file is corroboration after the name matched, never a reason on its own.Verification
CGO_ENABLED=1 go test -tags "fts5" ./... -count=1— all packages pass on the rebased basego vet ./...— cleangofmt -l .— only the two pre-existing parse fixtures undertestdata/binding_gap/go/baseline.jsonre-recorded with-update-golden; the diff is the four numbers above and nothing else🤖 Generated with Claude Code