Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions internal/app/search/rank/testdata/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,12 +313,32 @@ from it; the lists that also name `wiki_search` are dormant, as above.
never held a relevant answer, so the ranking was never given the chance and no
ranking change can fix that query.

### When a package node is a right answer

A package node is judged the same way any other node is: on the name the package
declares, matched the way the query's own bucket matches names. `mcp` judges the
node for the package that declares `package mcp`, because an exact-name query
judges an exact name; `annot` judges the package that declares `package
annotation`, because that query is a prefix and already counts `graph.Annotation`
a right answer on the same prefix.

Two things do not make a package node an answer. A path that happens to contain
the word does not: `internal/runtime/mcp` declares `package mcpruntime`, and a
directory spelled `mcp` is not the package's name. Holding files the key already
judges does not either. Where a `why` mentions that, it is corroboration after
the name already matched, never the reason on its own — otherwise every package
would inherit an answer from any one file inside it.

Judging a package node moves a number without touching code, so it lands under
the same rule as any other re-judgment: its own commit, no code beside it, and
the per-query movement written down.

## The two totals

```
bucket n retrieved Recall@10 top1 top3 MRR
ALL 86 75/86 0.744 (125/168) 47 62 0.644
ANSWERABLE 78 74/78 0.838 (124/148) 46 61 0.697
ALL 86 75/86 0.747 (127/170) 48 62 0.649
ANSWERABLE 78 74/78 0.840 (126/150) 47 61 0.703
```

That block is `make search-eval`'s own output for the `ccg` corpus, copied
Expand Down
10 changes: 5 additions & 5 deletions internal/app/search/rank/testdata/baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -72,18 +72,18 @@
"bucket": "prefix",
"retrieved": true,
"returned": 30,
"relevant": 4,
"found": 2,
"rank": 2,
"relevant": 5,
"found": 3,
"rank": 1,
"weak_filtered": 9
},
{
"query": "crossref",
"bucket": "prefix",
"retrieved": true,
"returned": 34,
"relevant": 3,
"found": 3,
"relevant": 4,
"found": 4,
"rank": 1,
"weak_filtered": 1
},
Expand Down
12 changes: 7 additions & 5 deletions internal/app/search/rank/testdata/queries.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,19 +80,21 @@
"class:graph.Annotation@internal/domain/graph/annotation.go",
"file:internal/domain/graph/annotation.go@internal/domain/graph/annotation.go",
"file:internal/domain/annotation/parser.go@internal/domain/annotation/parser.go",
"file:internal/domain/annotation/normalizer.go@internal/domain/annotation/normalizer.go"
"file:internal/domain/annotation/normalizer.go@internal/domain/annotation/normalizer.go",
"package:github.com/tae2089/code-context-graph/internal/domain/annotation@internal/domain/annotation"
],
"why": "Short prefix over a crowded namespace. The domain type or its own package files are what someone typing five letters wants, not an MCP response DTO."
"why": "Short prefix over a crowded namespace. The domain type or its own package files are what someone typing five letters wants, not an MCP response DTO. The package node is judged with them for consistency with the rest of this key rather than as a new kind of answer: this is a prefix query, and the key already accepts a prefix match on a name — graph.Annotation is judged though the query stops five letters short. The package declares the name annotation, so it matches the same way, and it is the directory holding two files this key already calls right answers."
},
{
"query": "crossref",
"bucket": "prefix",
"relevant": [
"class:graph.CrossRef@internal/domain/graph/crossref.go",
"file:internal/domain/graph/crossref.go@internal/domain/graph/crossref.go",
"file:internal/app/crossref/service.go@internal/app/crossref/service.go"
"file:internal/app/crossref/service.go@internal/app/crossref/service.go",
"package:github.com/tae2089/code-context-graph/internal/app/crossref@internal/app/crossref"
],
"why": "The domain type and the service that owns the feature outrank the persistence and transport shells."
"why": "The domain type and the service that owns the feature outrank the persistence and transport shells. The package node is judged on the same reasoning as impact: one word naming a whole feature, and the package declares that word exactly. It is also the directory holding service.go, which this key already names."
},
{
"query": "impact",
Expand Down Expand Up @@ -176,7 +178,7 @@
"relevant": [
"package:github.com/tae2089/code-context-graph/internal/adapters/inbound/mcp@internal/adapters/inbound/mcp"
],
"why": "A three-letter acronym naming a whole adapter. The package node is the answer; nothing smaller is. An earlier judgment also accepted mcp/deps.go, which cannot be defended: the pool holds about twenty-five files from that same package and deps.go is not distinguished from any of them. Judging one and not its siblings measured nothing. The package node is not in the candidate pool, so this query is now a retrieval finding, not a ranking one: no reordering can answer it while the only correct node never arrives."
"why": "A three-letter acronym naming a whole adapter. The package node is the answer; nothing smaller is. An earlier judgment also accepted mcp/deps.go, which cannot be defended: the pool holds about twenty-five files from that same package and deps.go is not distinguished from any of them. Judging one and not its siblings measured nothing. A second package node sits in the pool right behind it and is deliberately not judged: internal/runtime/mcp matches only because a directory in its path is spelled mcp, while the package it declares is named mcpruntime. The name a package declares is what this set judges on, not the path it happens to sit under."
},
{
"query": "cfg",
Expand Down
Loading