From e0e68d171215d013de586263c846d69766c94698 Mon Sep 17 00:00:00 2001 From: tae2089 Date: Tue, 11 Aug 2026 15:32:26 +0900 Subject: [PATCH] test: judge the package nodes annot and crossref already rank first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two queries name a package by the name that package declares and were judged on everything but the package node itself. `annot` is a prefix query whose key already accepts a prefix match on a name — `graph.Annotation` is judged though the query stops five letters short — and `internal/domain/annotation` declares `package annotation`. `crossref` names a whole feature in one word, and `internal/app/crossref` declares `package crossref`; `impact` was judged on that same reasoning already. `mcp` gains no node. Its `why` now records why the second package node in its pool stays unjudged: `internal/runtime/mcp` declares `package mcpruntime`, and only a directory in its path is spelled `mcp`. This is a re-judgment, so it moves numbers with no code beside it. Per query, measured by rerunning the report with each edit alone: annot relevant 4 -> 5, found 2 -> 3, first relevant rank 2 -> 1 crossref relevant 3 -> 4, found 3 -> 4, rank 1 unchanged mcp unchanged Only the prefix bucket moves: 0.882 (15/17) top1 4 MRR 0.900 becomes 0.895 (17/19) top1 5 MRR 1.000. ccg ALL 0.744 (125/168) 47 62 0.644 becomes 0.747 (127/170) 48 62 0.649; ANSWERABLE 0.838 (124/148) 46 61 0.697 becomes 0.840 (126/150) 47 61 0.703. cobra, context-diary and gorm are untouched. The top1 that annot regains is the one it lost when the package nodes were added to the pool: the annotation package node is what the ranker had already put at rank 1 while the key called it wrong. Nothing about the ranking changed here — only which nodes the key admits are right. testdata/README.md states the rule the three judgments follow, including its two negative cases: a path segment spelled like the query is not the package's name, and containing an already-judged file is corroboration after the name matched, never a reason on its own. Co-Authored-By: Claude Opus 5 --- internal/app/search/rank/testdata/README.md | 24 +++++++++++++++++-- .../app/search/rank/testdata/baseline.json | 10 ++++---- .../app/search/rank/testdata/queries.json | 12 ++++++---- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/internal/app/search/rank/testdata/README.md b/internal/app/search/rank/testdata/README.md index 988e0e8..0698e5d 100644 --- a/internal/app/search/rank/testdata/README.md +++ b/internal/app/search/rank/testdata/README.md @@ -313,12 +313,32 @@ from it; the lists that also name `wiki_search` are dormant, as above. never held a relevant answer, so the ranking was never given the chance and no ranking change can fix that query. +### When a package node is a right answer + +A package node is judged the same way any other node is: on the name the package +declares, matched the way the query's own bucket matches names. `mcp` judges the +node for the package that declares `package mcp`, because an exact-name query +judges an exact name; `annot` judges the package that declares `package +annotation`, because that query is a prefix and already counts `graph.Annotation` +a right answer on the same prefix. + +Two things do not make a package node an answer. A path that happens to contain +the word does not: `internal/runtime/mcp` declares `package mcpruntime`, and a +directory spelled `mcp` is not the package's name. Holding files the key already +judges does not either. Where a `why` mentions that, it is corroboration after +the name already matched, never the reason on its own — otherwise every package +would inherit an answer from any one file inside it. + +Judging a package node moves a number without touching code, so it lands under +the same rule as any other re-judgment: its own commit, no code beside it, and +the per-query movement written down. + ## The two totals ``` bucket n retrieved Recall@10 top1 top3 MRR -ALL 86 75/86 0.744 (125/168) 47 62 0.644 -ANSWERABLE 78 74/78 0.838 (124/148) 46 61 0.697 +ALL 86 75/86 0.747 (127/170) 48 62 0.649 +ANSWERABLE 78 74/78 0.840 (126/150) 47 61 0.703 ``` That block is `make search-eval`'s own output for the `ccg` corpus, copied diff --git a/internal/app/search/rank/testdata/baseline.json b/internal/app/search/rank/testdata/baseline.json index 6726574..c37450a 100644 --- a/internal/app/search/rank/testdata/baseline.json +++ b/internal/app/search/rank/testdata/baseline.json @@ -72,9 +72,9 @@ "bucket": "prefix", "retrieved": true, "returned": 30, - "relevant": 4, - "found": 2, - "rank": 2, + "relevant": 5, + "found": 3, + "rank": 1, "weak_filtered": 9 }, { @@ -82,8 +82,8 @@ "bucket": "prefix", "retrieved": true, "returned": 34, - "relevant": 3, - "found": 3, + "relevant": 4, + "found": 4, "rank": 1, "weak_filtered": 1 }, diff --git a/internal/app/search/rank/testdata/queries.json b/internal/app/search/rank/testdata/queries.json index cdc4a19..0a6aa35 100644 --- a/internal/app/search/rank/testdata/queries.json +++ b/internal/app/search/rank/testdata/queries.json @@ -80,9 +80,10 @@ "class:graph.Annotation@internal/domain/graph/annotation.go", "file:internal/domain/graph/annotation.go@internal/domain/graph/annotation.go", "file:internal/domain/annotation/parser.go@internal/domain/annotation/parser.go", - "file:internal/domain/annotation/normalizer.go@internal/domain/annotation/normalizer.go" + "file:internal/domain/annotation/normalizer.go@internal/domain/annotation/normalizer.go", + "package:github.com/tae2089/code-context-graph/internal/domain/annotation@internal/domain/annotation" ], - "why": "Short prefix over a crowded namespace. The domain type or its own package files are what someone typing five letters wants, not an MCP response DTO." + "why": "Short prefix over a crowded namespace. The domain type or its own package files are what someone typing five letters wants, not an MCP response DTO. The package node is judged with them for consistency with the rest of this key rather than as a new kind of answer: this is a prefix query, and the key already accepts a prefix match on a name — graph.Annotation is judged though the query stops five letters short. The package declares the name annotation, so it matches the same way, and it is the directory holding two files this key already calls right answers." }, { "query": "crossref", @@ -90,9 +91,10 @@ "relevant": [ "class:graph.CrossRef@internal/domain/graph/crossref.go", "file:internal/domain/graph/crossref.go@internal/domain/graph/crossref.go", - "file:internal/app/crossref/service.go@internal/app/crossref/service.go" + "file:internal/app/crossref/service.go@internal/app/crossref/service.go", + "package:github.com/tae2089/code-context-graph/internal/app/crossref@internal/app/crossref" ], - "why": "The domain type and the service that owns the feature outrank the persistence and transport shells." + "why": "The domain type and the service that owns the feature outrank the persistence and transport shells. The package node is judged on the same reasoning as impact: one word naming a whole feature, and the package declares that word exactly. It is also the directory holding service.go, which this key already names." }, { "query": "impact", @@ -176,7 +178,7 @@ "relevant": [ "package:github.com/tae2089/code-context-graph/internal/adapters/inbound/mcp@internal/adapters/inbound/mcp" ], - "why": "A three-letter acronym naming a whole adapter. The package node is the answer; nothing smaller is. An earlier judgment also accepted mcp/deps.go, which cannot be defended: the pool holds about twenty-five files from that same package and deps.go is not distinguished from any of them. Judging one and not its siblings measured nothing. The package node is not in the candidate pool, so this query is now a retrieval finding, not a ranking one: no reordering can answer it while the only correct node never arrives." + "why": "A three-letter acronym naming a whole adapter. The package node is the answer; nothing smaller is. An earlier judgment also accepted mcp/deps.go, which cannot be defended: the pool holds about twenty-five files from that same package and deps.go is not distinguished from any of them. Judging one and not its siblings measured nothing. A second package node sits in the pool right behind it and is deliberately not judged: internal/runtime/mcp matches only because a directory in its path is spelled mcp, while the package it declares is named mcpruntime. The name a package declares is what this set judges on, not the path it happens to sit under." }, { "query": "cfg",