Skip to content

test: judge the package nodes annot and crossref already rank first - #105

Merged
tae2089 merged 1 commit into
mainfrom
ticket-102
Aug 11, 2026
Merged

test: judge the package nodes annot and crossref already rank first#105
tae2089 merged 1 commit into
mainfrom
ticket-102

Conversation

@tae2089

@tae2089 tae2089 commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Closes #102. Rebased onto main after #101, #107 and #104 landed; the numbers below are remeasured on that base.

What this is

A re-judgment of the golden set, with no code beside it. Two queries name a package by the name that package declares and were judged on everything but the package node itself.

  • annot — prefix query. Its key already accepts a prefix match on a name (graph.Annotation is judged though the query stops five letters short), and internal/domain/annotation declares package annotation.
  • crossref — one word naming a whole feature. internal/app/crossref declares package crossref, and impact was judged on that same reasoning already.
  • mcp gains no node. Its why now records why the other package node in its pool stays unjudged: internal/runtime/mcp declares package mcpruntime, and only a directory in its path is spelled mcp.

A sweep of all four corpora found no other unjudged package node in any top-15 pool: cobra and context-diary have none in any pool at all, and gorm's clause is already judged.

Movement

Measured by rerunning the report with each edit alone:

annot     relevant 4 -> 5, found 2 -> 3, first relevant rank 2 -> 1
crossref  relevant 3 -> 4, found 3 -> 4, rank 1 unchanged
mcp       unchanged

Only the prefix bucket moves: 0.882 (15/17) top1 4 MRR 0.900 becomes 0.895 (17/19) top1 5 MRR 1.000 — the same movement as before the rebase; that bucket is untouched by the three PRs that landed in between.

ccg ALL         0.744 (125/168) 47 62 0.644  ->  0.747 (127/170) 48 62 0.649
ccg ANSWERABLE  0.838 (124/148) 46 61 0.697  ->  0.840 (126/150) 47 61 0.703

cobra, context-diary and gorm are untouched; no fixture was recaptured. testdata/README.md's ## The two totals block is updated to the new output, so it does not go stale again.

The top1 annot regains is the one it lost in #101 when package nodes entered the pool. The annotation package node is what the ranker had already put at rank 1 while the key called it wrong. Nothing about the ranking changed here — only which nodes the key admits are right.

Rule

testdata/README.md gains the rule these three judgments follow, with its two negative cases: a path segment spelled like the query is not the package's name, and containing an already-judged file is corroboration after the name matched, never a reason on its own.

Verification

  • CGO_ENABLED=1 go test -tags "fts5" ./... -count=1 — all packages pass on the rebased base
  • go vet ./... — clean
  • gofmt -l . — only the two pre-existing parse fixtures under testdata/binding_gap/go/
  • baseline.json re-recorded with -update-golden; the diff is the four numbers above and nothing else

🤖 Generated with Claude Code

@tae2089
tae2089 changed the base branch from ticket-96-v2 to main August 11, 2026 07:10
Two queries name a package by the name that package declares and were judged
on everything but the package node itself. `annot` is a prefix query whose key
already accepts a prefix match on a name — `graph.Annotation` is judged though
the query stops five letters short — and `internal/domain/annotation` declares
`package annotation`. `crossref` names a whole feature in one word, and
`internal/app/crossref` declares `package crossref`; `impact` was judged on
that same reasoning already.

`mcp` gains no node. Its `why` now records why the second package node in its
pool stays unjudged: `internal/runtime/mcp` declares `package mcpruntime`, and
only a directory in its path is spelled `mcp`.

This is a re-judgment, so it moves numbers with no code beside it. Per query,
measured by rerunning the report with each edit alone:

  annot     relevant 4 -> 5, found 2 -> 3, first relevant rank 2 -> 1
  crossref  relevant 3 -> 4, found 3 -> 4, rank 1 unchanged
  mcp       unchanged

Only the prefix bucket moves: 0.882 (15/17) top1 4 MRR 0.900 becomes
0.895 (17/19) top1 5 MRR 1.000. ccg ALL 0.744 (125/168) 47 62 0.644 becomes
0.747 (127/170) 48 62 0.649; ANSWERABLE 0.838 (124/148) 46 61 0.697 becomes
0.840 (126/150) 47 61 0.703. cobra, context-diary and gorm are untouched.

The top1 that annot regains is the one it lost when the package nodes were
added to the pool: the annotation package node is what the ranker had already
put at rank 1 while the key called it wrong. Nothing about the ranking changed
here — only which nodes the key admits are right.

testdata/README.md states the rule the three judgments follow, including its
two negative cases: a path segment spelled like the query is not the package's
name, and containing an already-judged file is corroboration after the name
matched, never a reason on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tae2089
tae2089 merged commit db49483 into main Aug 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge package nodes across the existing golden answer keys

1 participant