feat(search): adopt a real lexical index (BM25/FTS5) behind the search provider port

[drewstone]

The consumer pressure

A discovery-lab store now holds 1,093 verified claim pages and grows by ~50–100 per day. Its research directors are instructed to dedup new questions BY MEANING against the store before authoring, and its leads must kb_search before computing. Both run through filesystem-search-provider scoring, which is custom term matching rather than a ranked lexical index.

Measured symptoms on that store:

• TF-IDF cosine over page bodies (computed externally) finds 36 near-duplicate clusters (98 pages) the search surface does not expose.
• Retrieval quality is the binding input to question dedup: the field's measured result is that embedding-style similarity misses ~24% of expert-caught duplication, while idea-level retrieval beats textual similarity by +24pp (arXiv 2505.24615). A proper ranked lexical baseline is the floor under both.

The ask

Adopt a standard lexical index behind the existing search provider port — SQLite FTS5 (BM25 built in, zero new service, better-sqlite3 is already in the ecosystem) or a pure-JS BM25 (minisearch/wink) if a native dep is unwanted. Keep the provider interface; the index is an implementation.

Per the shared doctrine this belongs upstream: consumers should not each grow their own retrieval. The discovery lab will delete its external TF-IDF pass the day ask() ranks with BM25.

🤖 Generated with Claude Code