From 8dc3f3032606740dcebed943600ac3e53ee59d7a Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 15:36:05 +0100 Subject: [PATCH 1/9] DOC-6809 Add docs-only MCP server spec + v0 prototype MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigation into a read-only "docs MCP server" — distinct from the existing data-plane redis/mcp-redis — that exposes the documentation corpus as agent-queryable tools over the docs.ndjson feed we already publish. Spec plus a working v0 (search_docs + get_page, self-contained BM25, no datastore). Two findings from running it against the live feed shaped the design: indexing 2,531 pages takes ~0.3s and queries ~100ms, so search compute is a non-issue and Rust/WASM would be premature optimisation; but untuned lexical ranking mis-ranks natural-language queries — with no stemmer, "append" misses XADD's "appends", and canonical command pages lose to release-notes, operator custom-resources, and client-library pages that repeat the same terms. That ranking gap, not speed, is what justifies the vector-search upgrade path in the spec. Learned: measured perf rules out WASM; lexical ranking (not speed) is the real limit Constraint: docs MCP server stays read-only, no DB connection, no creds (unlike mcp-redis data plane) Rejected: Rust/WASM for search speed | ~0.3s index + ~100ms queries on 2,531 pages, compute isn't the bottleneck Directive: don't over-tune lexical BM25 weights (overfits to sample queries); fix ranking via stemming/analyzer or vector search Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 232 ++++++++++++++++++ build/docs-mcp-server/node/.gitignore | 3 + build/docs-mcp-server/node/README.md | 78 ++++++ build/docs-mcp-server/node/package.json | 31 +++ build/docs-mcp-server/node/src/feed.ts | 44 ++++ build/docs-mcp-server/node/src/index.ts | 111 +++++++++ build/docs-mcp-server/node/src/search.ts | 198 +++++++++++++++ build/docs-mcp-server/node/src/smoke.ts | 28 +++ .../node/src/tools/get-page.ts | 40 +++ .../node/src/tools/search-docs.ts | 20 ++ build/docs-mcp-server/node/src/types.ts | 35 +++ .../docs-mcp-server/node/test/fixture.ndjson | 3 + build/docs-mcp-server/node/tsconfig.json | 19 ++ 13 files changed, 842 insertions(+) create mode 100644 build/docs-mcp-server/SPEC.md create mode 100644 build/docs-mcp-server/node/.gitignore create mode 100644 build/docs-mcp-server/node/README.md create mode 100644 build/docs-mcp-server/node/package.json create mode 100644 build/docs-mcp-server/node/src/feed.ts create mode 100644 build/docs-mcp-server/node/src/index.ts create mode 100644 build/docs-mcp-server/node/src/search.ts create mode 100644 build/docs-mcp-server/node/src/smoke.ts create mode 100644 build/docs-mcp-server/node/src/tools/get-page.ts create mode 100644 build/docs-mcp-server/node/src/tools/search-docs.ts create mode 100644 build/docs-mcp-server/node/src/types.ts create mode 100644 build/docs-mcp-server/node/test/fixture.ndjson create mode 100644 build/docs-mcp-server/node/tsconfig.json diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md new file mode 100644 index 0000000000..3a8a739d21 --- /dev/null +++ b/build/docs-mcp-server/SPEC.md @@ -0,0 +1,232 @@ +# Docs MCP server — design spec + +**Status:** Draft (investigation, DOC-6809) +**Owner:** Docs +**Scope:** A read-only MCP server that lets AI coding agents query the Redis +documentation corpus as a tool, backed by the JSON/NDJSON feed we already +publish. + +--- + +## 1. Motivation + +We already publish AI-readable docs three ways: `llms.txt` (index), per-page +Markdown (`index.html.md`), and structured JSON/NDJSON with role-tagged +sections. These are all *passive files* — an agent must know they exist, fetch +them, and do its own retrieval. + +The gap is an *active, queryable* surface: a tool an agent (Cursor, Claude +Code, ChatGPT, VS Code) can call mid-task to get a **current, sourced** answer +instead of relying on stale training data. That is what this server provides. + +### Explicitly *not* this server + +- **`redis/mcp-redis`** is a *data-plane* server: it connects an agent to a + *running Redis instance* to read/write/query data. It needs a connection + string and can mutate data. +- **This server** is a *knowledge-plane* server: it connects an agent to the + *documentation*. It is read-only, needs no database, no credentials, and its + entire value is returning citations to our docs. + +They are orthogonal and should stay separate products/installs. Bundling doc +lookup into the data-plane server forces the reference-only audience to stand +up a data server and hand it credentials — friction that kills adoption for +the exact audience (coding agents) that benefits most. + +## 2. Goals / non-goals + +**Goals** +- Thin retrieval wrapper over the **existing** JSON feed — no new content + pipeline. +- Read-only, no secrets, no live DB connection. +- Every response carries a canonical `url` (citations by construction). +- Token-lean: search returns summaries + refs; agents drill down deliberately. +- Version-aware (our docs are versioned; mixing versions is a correctness bug). + +**Non-goals** +- No writes, no code execution, no live Redis access (that's `mcp-redis`). +- No new authoring format — we consume `sections[]` / `examples[]` as-is. +- WebMCP / in-browser tool registration — out of scope for now (different + layer, different audience; see DOC-6809 discussion). + +## 3. Data source + +No new pipeline. Reuse the current build output: + +``` +Hugo ──► per-page public/**/index.json ──► generate_ndjson.py ──► docs.ndjson +``` + +Document schema (already published on the *AI Agent Resources* page): + +- **Page**: `id`, `title`, `url`, `summary`, `page_type` (`content` | `index`), + `content_hash`, `sections[]`, `examples[]`, `children[]` +- **Section**: `id`, `title`, `role` (`overview` | `syntax` | `parameters` | + `returns` | `example` | …), `text` +- **Example**: `id`, `language`, `code`, `section_id` + +The server loads `docs.ndjson` (or an index built from it) at startup. Because +`content_hash` is deterministic (`sha256` over summary + section text + +example code), it doubles as a cache/freshness key. + +## 4. Tool surface + +Five tools. Names, inputs, and the field of the existing schema each is +projected from: + +### `search_docs` +Rank pages by relevance. Returns **refs only, no full text**. + +- **In:** `query` (string, required); optional `page_type`, `group`, + `version`, `limit` (default 10) +- **Out:** `[{ id, title, url, summary, matching_section_ids[] }]` +- **From:** NDJSON feed; `page_type` filter lets callers skip `index` pages. + +### `get_page` +Fetch one page, optionally filtered to specific section roles. + +- **In:** `id` **or** `url` (required); optional `roles[]` (e.g. + `["syntax","parameters"]`) +- **Out:** `{ id, title, url, summary, page_type, content_hash, sections[] }` + where `sections` is filtered to `roles[]` if given +- **From:** per-page `index.json`. `roles[]` filtering is only possible because + sections are role-tagged — big token savings (pull `parameters` without the + overview prose). + +### `get_section` +Return a single role-tagged chunk — the retrieval-native unit. + +- **In:** `page_id` (required), `section_id` (required) +- **Out:** `{ page_id, section_id, title, role, text, url }` +- **From:** `sections[]`. + +### `get_examples` +Return runnable code, filterable by language. **The highest-value tool for +coding agents** — their most common need is "the go-redis snippet for `XADD`", +not prose. + +- **In:** `query` **or** `command` (one required); optional `language` + (`python` | `go` | `java` | …) +- **Out:** `[{ id, code, language, url, section_id }]` +- **From:** `examples[]` (carries `language` + `section_id` already). + +### `get_command` +Convenience lookup for the highest-traffic page type. + +- **In:** `name` (e.g. `XADD`) +- **Out:** command page with `syntax`, `parameters`, `returns` sections + + `examples[]` + `url` +- **From:** command pages (specialised `get_page`; commands are first-class). + +## 5. Response conventions + +- **Always include `url`.** Agents cite; users click. +- **Search never returns full text.** Force the drill-down path + (`search_docs` → `get_section` / `get_examples`) so context stays small. +- **Return `content_hash` on page/section responses.** Lets agents and our own + eval harness do `If-None-Match`-style freshness checks for free. +- **Truncate defensively.** Cap `text` length per section in responses; expose + a `truncated: true` flag rather than silently cutting. + +## 6. Transport & deployment + +Follow the existing repo pattern (`build/command_api_mapping/mcp-server/`): +TypeScript, `@modelcontextprotocol/sdk`, Zod input schemas. + +- **Local / stdio:** publish an npx-runnable package so developers can add it to + Cursor/Claude Code config. Zero infra. +- **Remote / hosted:** an HTTP+SSE endpoint on `redis.io` (e.g. + `https://redis.io/mcp`) built from the same handlers, so no install is + required. Advertise it on the *AI Agent Resources* page next to `llms.txt`. + +Both modes share one core: load feed → build index → handle tool calls. The +data is public, so the remote endpoint needs no auth (rate-limit only). + +### Search backend vs. transport — what needs a datastore + +Whether the server needs a Redis (or any) backend depends entirely on the +search implementation, **not** on the transport. The corpus is small +(`docs.ndjson` ≈ 30 MB / ≈ 5 MB gzipped, ≈ 4,100 docs), which is what makes +the lexical path infra-free. + +| | Lexical (BM25) | Vector (semantic) | +|---|---|---| +| **stdio (client-side)** | ✅ self-contained in-memory index | ❌ impractical (would ship an index + embedding model per install) | +| **remote (hosted)** | ✅ in-process index, no datastore | ✅ needs a vector store (RediSearch / RedisVL) | + +- **Lexical (v1):** build a BM25 index in process memory from `docs.ndjson` at + startup — MiniSearch/Lunr (JS) or a simple BM25 (Python). No datastore, even + when hosted; horizontally-scaled instances each just load ~5 MB gzipped at + boot and build their own index. Role-tagged sections and exact command-name + tokens make lexical retrieval unusually strong on this corpus. +- **Vector (v2, hosted only):** needs embeddings computed offline at build + time, a vector index queried at runtime, and query-time embedding on each + request. That implies a real server-side backend — the natural fit is + **RediSearch / RedisVL**, which doubles as a Redis showcase. It cannot run + purely client-side, so it's a hosted-endpoint upgrade, not a stdio feature. + +Recommendation: ship v1 lexical in **both** modes with no backend; treat +vector-on-Redis as a later upgrade to the **hosted** endpoint only. + +## 7. Versioning + +- `search_docs` / `get_page` accept an optional `version`; default to `latest`. +- Responses echo the resolved version so an agent can't silently blend + versions. This is the single most common RAG-over-docs correctness bug. + +## 8. Freshness + +- Rebuild the server's index whenever `docs.ndjson` is regenerated (same build + step). No separate content pipeline to keep in sync. +- `content_hash` per page enables incremental index updates and client caching. + +## 9. Security + +- Read-only. No write tools, no code execution, no connection string. +- Serves only already-public content. +- Remote endpoint: rate-limit, no auth, no PII. + +## 10. Open questions + +- **Search backend:** resolved for v1 — lexical (BM25 over NDJSON), no + datastore, runs in both stdio and hosted modes (see §6). Open: do we add + vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / + RedisVL, and is the retrieval gain worth the added infra given how + well-structured the corpus already is? +- **Ranking quality (found in v0 prototype):** untuned lexical BM25 — even + with stopword removal and title/summary/slug field boosts — mis-ranks + natural-language queries. Two concrete failure modes observed on the live + feed: (1) **no stemming**, so "append" ≠ "appends" and `XADD` won't surface + for "append an entry to a stream"; (2) **canonical command pages compete with + release notes, operator `custom-resources` pages, and client-library + overviews** that repeat the same terms. Fix needs an analyzer + (stemming/lemmatization), possibly a page-type/canonical boost, and/or the + vector-search path. This is the strongest argument for §6's v2 upgrade. +- **`get_command` coverage:** command pages *do* carry `parameters`/`returns`/ + `example` roles (confirmed: `get_page('expire')` → roles + `[content, parameters, example, returns]`). Confirm this holds across all + command pages or add a fallback. +- **Package ownership:** does this live here in `docs`, or graduate to its own + repo like `mcp-redis`? +- **Overlap with `mcp-redis`:** does that server already do any doc lookup we + should pull into here and deprecate there? + +## 11. Phased plan + +1. **v0 (prototype):** stdio server, lexical `search_docs` + `get_page` over a + local `docs.ndjson`. Prove the loop in Claude Code. +2. **v1:** add `get_examples`, `get_section`, `get_command`; `version` support; + token-budget guards. +3. **v2:** hosted remote endpoint on `redis.io`; advertise on AI Agent + Resources. +4. **Ongoing:** wire an **AI-answer eval** — a fixed set of real questions run + through the server, scored for correctness — as a docs-quality regression + gate. (Extends the code-example verification mindset to answer quality.) + +## 12. Success signals + +- A coding agent, given only this server, answers common Redis how-to questions + correctly and with citations. +- Measurable reduction in version-mixing / stale-API answers versus the model's + own training data. +- Adoption: entries in Cursor/Claude Code MCP configs pointing at the endpoint. diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore new file mode 100644 index 0000000000..3c25e1e49c --- /dev/null +++ b/build/docs-mcp-server/node/.gitignore @@ -0,0 +1,3 @@ +node_modules/ +dist/ +*.log diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md new file mode 100644 index 0000000000..0bd4af5209 --- /dev/null +++ b/build/docs-mcp-server/node/README.md @@ -0,0 +1,78 @@ +# redis-docs-mcp (v0 prototype) + +A read-only MCP server that lets an AI coding agent query the Redis +documentation corpus as a tool, backed by the `docs.ndjson` feed we already +publish. See [`../SPEC.md`](../SPEC.md) for the full design. + +**This is a v0 prototype.** It ships two tools (`search_docs`, `get_page`) over +an in-memory lexical (BM25 + field-boost) index. No datastore, no credentials, +no live Redis connection. + +## Install & build + +```bash +cd build/docs-mcp-server/node +npm install +npm run build # compiles to dist/ +``` + +## Try it offline (no network) + +```bash +npm run smoke # runs against test/fixture.ndjson +``` + +Point it at the real feed (or any local `.ndjson` / `.ndjson.gz`): + +```bash +DOCS_NDJSON="https://redis.io/docs/latest/docs.ndjson" npm run smoke +``` + +## Run as an MCP server + +The server speaks MCP over stdio. Feed source is set via `DOCS_NDJSON` +(default: `https://redis.io/docs/latest/docs.ndjson`). + +Add to a Claude Code / Cursor MCP config after `npm run build`: + +```json +{ + "mcpServers": { + "redis-docs": { + "command": "node", + "args": ["/ABSOLUTE/PATH/build/docs-mcp-server/node/dist/index.js"], + "env": { "DOCS_NDJSON": "https://redis.io/docs/latest/docs.ndjson" } + } + } +} +``` + +## Tools + +| Tool | Inputs | Returns | +|------|--------|---------| +| `search_docs` | `query` (req), `page_type`, `version`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | +| `get_page` | `id` **or** `url` (req), `roles[]` | one page with `content_hash` + `sections`, optionally filtered to the given section roles | + +Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` +(e.g. `["parameters","returns"]`) to pull just what's needed. + +## Measured (real feed, 2,531 pages) + +- Index build: ~0.3 s. Query latency: ~75–125 ms. This is why v0 needs no + datastore and why Rust/WASM would be premature (see SPEC §6). + +## Known limitations (v0) + +- **Ranking is untuned lexical search.** Good on distinctive terms + (`publish`, `incrby`, `pexpire`), but weaker where: + - there is **no stemmer** — a query for "append" won't match a summary that + says "appends", so `XADD` is hard to surface from natural language; + - **canonical command pages** compete with release notes, operator + (custom-resources) pages, and client-library overviews that repeat the same + terms. + Fixing this properly means an analyzer (stemming/lemmatization) and/or the + vector-search upgrade tracked in SPEC §6/§10. +- **Version filtering is heuristic** (URL-path based); see SPEC §7. +- Only `search_docs` + `get_page`. `get_examples`, `get_section`, + `get_command` are v1 (SPEC §4/§11). diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json new file mode 100644 index 0000000000..ce615096b0 --- /dev/null +++ b/build/docs-mcp-server/node/package.json @@ -0,0 +1,31 @@ +{ + "name": "redis-docs-mcp", + "version": "0.0.1", + "description": "Read-only MCP server that queries the Redis documentation corpus (docs.ndjson) as a tool. v0 prototype: lexical search_docs + get_page.", + "type": "module", + "main": "dist/index.js", + "bin": { + "redis-docs-mcp": "dist/index.js" + }, + "scripts": { + "build": "tsc", + "start": "tsx src/index.ts", + "dev": "tsx watch src/index.ts", + "smoke": "tsx src/smoke.ts" + }, + "keywords": [ + "redis", + "mcp", + "docs" + ], + "license": "MIT", + "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "zod": "^3.22.0" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "tsx": "^4.0.0", + "typescript": "^5.0.0" + } +} diff --git a/build/docs-mcp-server/node/src/feed.ts b/build/docs-mcp-server/node/src/feed.ts new file mode 100644 index 0000000000..7eb01a0a78 --- /dev/null +++ b/build/docs-mcp-server/node/src/feed.ts @@ -0,0 +1,44 @@ +import { readFile } from "node:fs/promises"; +import { gunzipSync } from "node:zlib"; +import type { Page } from "./types.js"; + +/** Load raw feed bytes from a local path or http(s) URL, gunzipping if .gz. */ +async function loadFeedRaw(source: string): Promise { + let buf: Buffer; + if (/^https?:\/\//i.test(source)) { + const res = await fetch(source); + if (!res.ok) { + throw new Error(`Failed to fetch feed ${source}: ${res.status} ${res.statusText}`); + } + buf = Buffer.from(await res.arrayBuffer()); + } else { + buf = await readFile(source); + } + if (source.toLowerCase().endsWith(".gz")) { + buf = gunzipSync(buf); + } + return buf.toString("utf-8"); +} + +/** Parse NDJSON text into pages, skipping blank/invalid lines and non-doc objects. */ +export function parseNdjson(text: string): Page[] { + const pages: Page[] = []; + for (const line of text.split("\n")) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const obj = JSON.parse(trimmed); + // Match the same guard generate_ndjson.py uses: must have id + title + url. + if (obj && typeof obj.id === "string" && typeof obj.url === "string" && typeof obj.title === "string") { + pages.push(obj as Page); + } + } catch { + // Not our format — skip. + } + } + return pages; +} + +export async function loadFeed(source: string): Promise { + return parseNdjson(await loadFeedRaw(source)); +} diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts new file mode 100644 index 0000000000..ff6fbd87db --- /dev/null +++ b/build/docs-mcp-server/node/src/index.ts @@ -0,0 +1,111 @@ +#!/usr/bin/env node +import { Server } from "@modelcontextprotocol/sdk/server/index.js"; +import { + ListToolsRequestSchema, + CallToolRequestSchema, +} from "@modelcontextprotocol/sdk/types.js"; +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; + +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; +import { getPage, GetPageInput } from "./tools/get-page.js"; + +// Feed source: local path or http(s) URL, gzip-aware. Defaults to production. +const FEED_SOURCE = + process.env.DOCS_NDJSON ?? "https://redis.io/docs/latest/docs.ndjson"; + +const TOOLS = [ + { + name: "search_docs", + description: + "Search the Redis documentation and return the most relevant pages as references (id, title, url, summary, matching section ids). Returns no full text — follow up with get_page to read a result.", + inputSchema: { + type: "object" as const, + properties: { + query: { type: "string", description: "Search query" }, + page_type: { + type: "string", + enum: ["content", "index"], + description: "Restrict to prose ('content') or navigation ('index') pages", + }, + version: { + type: "string", + description: "Redis docs version, e.g. 'latest' (default) or '7.4'", + }, + limit: { + type: "number", + description: "Max results (default 10, max 50)", + }, + }, + required: ["query"], + }, + }, + { + name: "get_page", + description: + "Fetch a single documentation page by 'id' or 'url'. Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", + inputSchema: { + type: "object" as const, + properties: { + id: { type: "string", description: "Page id (URL slug), e.g. 'commands/xadd'" }, + url: { type: "string", description: "Full or partial page URL" }, + roles: { + type: "array", + items: { type: "string" }, + description: "Only return sections with these roles", + }, + }, + }, + }, +]; + +function ok(data: unknown) { + return { content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }] }; +} + +function fail(message: string) { + return { + content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], + isError: true, + }; +} + +async function main() { + const pages = await loadFeed(FEED_SOURCE); + const index = new DocsIndex(pages); + // Log to stderr — stdout is reserved for the MCP protocol. + console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); + + const server = new Server( + { name: "redis-docs-mcp", version: "0.0.1" }, + { capabilities: { tools: {} } }, + ); + + server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS })); + + server.setRequestHandler(CallToolRequestSchema, async (req) => { + const { name, arguments: args } = req.params; + try { + switch (name) { + case "search_docs": + return ok(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + case "get_page": + return ok(getPage(index, GetPageInput.parse(args ?? {}))); + default: + return fail(`Unknown tool: ${name}`); + } + } catch (e) { + return fail(e instanceof Error ? e.message : String(e)); + } + }); + + const transport = new StdioServerTransport(); + await server.connect(transport); + console.error("[redis-docs-mcp] ready on stdio"); +} + +main().catch((e) => { + console.error("[redis-docs-mcp] fatal:", e); + process.exit(1); +}); diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts new file mode 100644 index 0000000000..d177d0556a --- /dev/null +++ b/build/docs-mcp-server/node/src/search.ts @@ -0,0 +1,198 @@ +import type { Page } from "./types.js"; + +// Self-contained BM25 lexical index. No external search dependency: at +// ~4,100 docs the whole index builds in-memory in well under a second, which +// is why v0 needs no datastore (see SPEC.md §6). + +const K1 = 1.5; +const B = 0.75; + +// Field boosts (added on top of the body BM25 score, weighted by term idf). +// A query term appearing in the title/slug/summary is a strong signal that the +// page is *about* that term — this lifts canonical command pages (whose summary +// is a one-line definition) above long pages that merely mention the terms. +const W_SUMMARY = 6; // canonical one-line definition — strongest signal +const W_TITLE = 4; +const W_SLUG = 2; // lowest: slug word-collisions (set-up-redis, key-specs) mislead + +// Common words carry no topical signal and their title/slug/summary collisions +// distort ranking (e.g. "set"/"key"). Dropped from the query only. +const STOPWORDS = new Set([ + "a", "an", "the", "to", "of", "in", "on", "for", "with", "and", "or", "is", + "are", "how", "do", "i", "my", "me", "can", "what", "when", "which", "you", + "your", "it", "this", "that", "from", "by", "as", "at", "be", "using", "use", +]); + +/** Split on non-alphanumerics so "JSON.SET" -> ["json","set"], "XADD" -> ["xadd"]. */ +function tokenize(text: string): string[] { + return text.toLowerCase().match(/[a-z0-9]+/g) ?? []; +} + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +/** Everything worth matching against for a page: slug, title, summary, section text. */ +function searchableText(p: Page): string { + const parts: string[] = [p.id ?? "", p.title ?? "", p.summary ?? ""]; + for (const s of p.sections ?? []) { + parts.push(s.title ?? "", s.text ?? ""); + } + return parts.join(" "); +} + +/** Section ids whose title/text contain any query term (capped for token budget). */ +function matchingSections(p: Page, qterms: Set): string[] { + const out: string[] = []; + for (const s of p.sections ?? []) { + const toks = new Set(tokenize(`${s.title ?? ""} ${s.text ?? ""}`)); + for (const t of qterms) { + if (toks.has(t)) { + out.push(s.id); + break; + } + } + if (out.length >= 5) break; + } + return out; +} + +/** + * Heuristic version filter. Redis docs carry the version in the URL path + * (e.g. /docs/latest/... or /docs/7.4/...). "latest"/unset does not filter. + * NOTE: approximate pending confirmation of how versions appear in the feed + * (SPEC.md §7 / open question). + */ +function matchesVersion(page: Page, version: string): boolean { + if (!version || version.toLowerCase() === "latest") return true; + return page.url.includes(`/${version}/`); +} + +export interface SearchOptions { + limit?: number; + pageType?: string; + version?: string; +} + +export interface SearchHit { + id: string; + title: string; + url: string; + summary: string; + page_type: string; + score: number; + matching_section_ids: string[]; +} + +export class DocsIndex { + readonly pages: Page[]; + private byId = new Map(); + private byUrl = new Map(); + private docs: Array<{ + page: Page; + tf: Map; + len: number; + titleTok: Set; + slugTok: Set; + summaryTok: Set; + }> = []; + private df = new Map(); + private avgdl = 0; + private N = 0; + + constructor(pages: Page[]) { + this.pages = pages; + let totalLen = 0; + for (const p of pages) { + this.byId.set(p.id, p); + if (p.url) this.byUrl.set(normalizeUrl(p.url), p); + + const tokens = tokenize(searchableText(p)); + if (tokens.length === 0) continue; + const tf = new Map(); + for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); + for (const t of tf.keys()) this.df.set(t, (this.df.get(t) ?? 0) + 1); + this.docs.push({ + page: p, + tf, + len: tokens.length, + titleTok: new Set(tokenize(p.title ?? "")), + slugTok: new Set(tokenize(p.id ?? "")), + summaryTok: new Set(tokenize(p.summary ?? "")), + }); + totalLen += tokens.length; + } + this.N = this.docs.length; + this.avgdl = this.N ? totalLen / this.N : 0; + } + + get size(): number { + return this.pages.length; + } + + getById(id: string): Page | undefined { + return this.byId.get(id); + } + + getByUrl(url: string): Page | undefined { + return this.byUrl.get(normalizeUrl(url)); + } + + /** Fallback lookup when a caller passes a path or partial URL. */ + findByUrlSuffix(url: string): Page | undefined { + const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); + if (!target) return undefined; + for (const p of this.pages) { + if (normalizeUrl(p.url).endsWith(target)) return p; + } + return undefined; + } + + search(query: string, opts: SearchOptions = {}): SearchHit[] { + let qterms = [...new Set(tokenize(query))].filter((t) => !STOPWORDS.has(t)); + // If the query was *all* stopwords, fall back to using them rather than + // returning nothing. + if (qterms.length === 0) qterms = [...new Set(tokenize(query))]; + if (qterms.length === 0) return []; + + const idf = new Map(); + for (const t of qterms) { + const df = this.df.get(t) ?? 0; + idf.set(t, Math.log(1 + (this.N - df + 0.5) / (df + 0.5))); + } + + const qset = new Set(qterms); + const hits: SearchHit[] = []; + for (const d of this.docs) { + if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; + if (opts.version && !matchesVersion(d.page, opts.version)) continue; + + let score = 0; + for (const t of qterms) { + const termIdf = idf.get(t) ?? 0; + const tf = d.tf.get(t); + if (tf) { + const denom = tf + K1 * (1 - B + B * (d.len / (this.avgdl || 1))); + score += termIdf * ((tf * (K1 + 1)) / denom); + } + // Field boosts: reward the term appearing in high-signal fields. + if (d.slugTok.has(t)) score += termIdf * W_SLUG; + if (d.titleTok.has(t)) score += termIdf * W_TITLE; + if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; + } + if (score > 0) { + hits.push({ + id: d.page.id, + title: d.page.title, + url: d.page.url, + summary: d.page.summary ?? "", + page_type: d.page.page_type ?? "content", + score: Number(score.toFixed(4)), + matching_section_ids: matchingSections(d.page, qset), + }); + } + } + hits.sort((a, b) => b.score - a.score); + return hits.slice(0, opts.limit ?? 10); + } +} diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts new file mode 100644 index 0000000000..feef8ac118 --- /dev/null +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -0,0 +1,28 @@ +// Offline smoke test: exercises the index + tools directly (no MCP transport). +// npm run smoke # uses test/fixture.ndjson +// DOCS_NDJSON=... npm run smoke # point at a real feed (path or URL) +import { fileURLToPath } from "node:url"; +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs } from "./tools/search-docs.js"; +import { getPage } from "./tools/get-page.js"; + +const feed = + process.env.DOCS_NDJSON ?? + fileURLToPath(new URL("../test/fixture.ndjson", import.meta.url)); + +const pages = await loadFeed(feed); +const index = new DocsIndex(pages); +console.log(`loaded ${index.size} pages from ${feed}\n`); + +console.log("# search_docs('append an entry to a stream')"); +console.log(JSON.stringify(searchDocs(index, { query: "append an entry to a stream" }), null, 2)); + +console.log("\n# search_docs('store json document', page_type=content)"); +console.log(JSON.stringify(searchDocs(index, { query: "store json document", page_type: "content" }), null, 2)); + +console.log("\n# get_page(id='commands/xadd', roles=['parameters'])"); +console.log(JSON.stringify(getPage(index, { id: "commands/xadd", roles: ["parameters"] }), null, 2)); + +console.log("\n# get_page(url='/commands/xadd/') -- partial-URL fallback"); +console.log(JSON.stringify(getPage(index, { url: "/commands/xadd/", roles: ["return"] }).url ?? "not found", null, 2)); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts new file mode 100644 index 0000000000..96b5fbba59 --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -0,0 +1,40 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; + +export const GetPageInput = z + .object({ + id: z.string().optional(), + url: z.string().optional(), + roles: z.array(z.string()).optional(), + }) + .refine((v) => Boolean(v.id || v.url), { + message: "Provide either 'id' or 'url'.", + }); +export type GetPageInput = z.infer; + +/** Fetch one page, optionally filtered to sections with the given roles. */ +export function getPage(index: DocsIndex, input: GetPageInput) { + let page = input.id ? index.getById(input.id) : undefined; + if (!page && input.url) { + page = index.getByUrl(input.url) ?? index.findByUrlSuffix(input.url); + } + if (!page) { + return { error: `Page not found for ${JSON.stringify(input.id ?? input.url)}` }; + } + + let sections = page.sections ?? []; + if (input.roles && input.roles.length) { + const want = new Set(input.roles.map((r) => r.toLowerCase())); + sections = sections.filter((s) => want.has((s.role ?? "").toLowerCase())); + } + + return { + id: page.id, + title: page.title, + url: page.url, + summary: page.summary ?? "", + page_type: page.page_type ?? "content", + content_hash: page.content_hash, + sections, + }; +} diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts new file mode 100644 index 0000000000..32313207ed --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -0,0 +1,20 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; + +export const SearchDocsInput = z.object({ + query: z.string().min(1, "query is required"), + page_type: z.enum(["content", "index"]).optional(), + version: z.string().optional(), + limit: z.number().int().positive().max(50).optional(), +}); +export type SearchDocsInput = z.infer; + +/** Rank pages by relevance. Returns refs + summaries only — never full text. */ +export function searchDocs(index: DocsIndex, input: SearchDocsInput) { + const results = index.search(input.query, { + limit: input.limit ?? 10, + pageType: input.page_type, + version: input.version, + }); + return { query: input.query, count: results.length, results }; +} diff --git a/build/docs-mcp-server/node/src/types.ts b/build/docs-mcp-server/node/src/types.ts new file mode 100644 index 0000000000..82ab8cc32d --- /dev/null +++ b/build/docs-mcp-server/node/src/types.ts @@ -0,0 +1,35 @@ +// Document schema as published in docs.ndjson / per-page index.json. +// See content/ai-agent-resources.md for the authoritative field reference. + +export interface Section { + id: string; + title: string; + /** Semantic role: overview | syntax | parameters | returns | example | ... */ + role?: string; + text: string; +} + +export interface Example { + id: string; + language: string; + code: string; + section_id: string; +} + +export interface Child { + title?: string; + url?: string; +} + +export interface Page { + id: string; + title: string; + url: string; + summary?: string; + /** "content" (has prose) or "index" (navigation only) */ + page_type?: string; + content_hash?: string; + sections?: Section[]; + examples?: Example[]; + children?: Child[]; +} diff --git a/build/docs-mcp-server/node/test/fixture.ndjson b/build/docs-mcp-server/node/test/fixture.ndjson new file mode 100644 index 0000000000..6e0c4ca8b8 --- /dev/null +++ b/build/docs-mcp-server/node/test/fixture.ndjson @@ -0,0 +1,3 @@ +{"id":"commands/xadd","title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/","summary":"Appends a new entry to a stream.","page_type":"content","content_hash":"abc123","sections":[{"id":"overview","title":"XADD","role":"overview","text":"Appends the specified stream entry to the stream at the specified key. If the key does not exist, a new stream is created."},{"id":"parameters","title":"Parameters","role":"parameters","text":"key: the name of the stream. NOMKSTREAM: optional, do not create the stream if it does not exist. field value: one or more field-value pairs to add to the stream entry."},{"id":"return","title":"Return value","role":"returns","text":"Returns the ID of the added stream entry."}],"examples":[{"id":"overview-ex0","language":"python","code":"r.xadd('mystream', {'field': 'value'})","section_id":"overview"}],"children":[]} +{"id":"develop/data-types/json","title":"JSON","url":"https://redis.io/docs/latest/develop/data-types/json/","summary":"Store and query JSON documents in Redis.","page_type":"content","content_hash":"def456","sections":[{"id":"overview","title":"Redis JSON","role":"overview","text":"The JSON data type lets you store, update, and retrieve JSON values in a Redis database."},{"id":"example","title":"Example","role":"example","text":"Use JSON.SET to store a document and JSON.GET to retrieve values from it by path."}],"examples":[{"id":"example-ex0","language":"python","code":"r.json().set('doc', '$', {'a': 1})","section_id":"example"}],"children":[]} +{"id":"commands","title":"Commands","url":"https://redis.io/docs/latest/commands/","summary":"Reference documentation for all Redis commands.","page_type":"index","children":[{"title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/"}]} diff --git a/build/docs-mcp-server/node/tsconfig.json b/build/docs-mcp-server/node/tsconfig.json new file mode 100644 index 0000000000..01b2904d2f --- /dev/null +++ b/build/docs-mcp-server/node/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "esnext", + "lib": ["ES2022"], + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true, + "declaration": true, + "sourceMap": true, + "moduleResolution": "node" + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist", "src/smoke.ts"] +} From ea1b9725482f9a4ce9324baf2c771e47fcf7b4fe Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:05:53 +0100 Subject: [PATCH 2/9] DOC-6809 Fix get_page id/url ambiguity and error signalling (Bugbot + Codex) Address Cursor Bugbot findings on PR #3585 plus a Codex follow-up review. The feed's `id` is the last URL path segment and is NOT unique (~213 ids map to several pages), so `byId` is now a multimap and `get_page` resolves by the unique `url` first; an ambiguous `id` or partial `url` returns candidate urls instead of silently returning the wrong page. The MCP handler now sets `isError` when a tool result carries an `error`. Smoke test rewritten as assertions (incl. a colliding id="install" fixture pair) so this can't regress. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/src/index.ts | 30 +++++----- build/docs-mcp-server/node/src/response.ts | 24 ++++++++ build/docs-mcp-server/node/src/search.ts | 30 ++++++---- build/docs-mcp-server/node/src/smoke.ts | 56 +++++++++++++++---- .../node/src/tools/get-page.ts | 39 +++++++++++-- .../docs-mcp-server/node/test/fixture.ndjson | 2 + 6 files changed, 138 insertions(+), 43 deletions(-) create mode 100644 build/docs-mcp-server/node/src/response.ts diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index ff6fbd87db..43ac439c42 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -10,6 +10,7 @@ import { loadFeed } from "./feed.js"; import { DocsIndex } from "./search.js"; import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; import { getPage, GetPageInput } from "./tools/get-page.js"; +import { toolResult, fail } from "./response.js"; // Feed source: local path or http(s) URL, gzip-aware. Defaults to production. const FEED_SOURCE = @@ -19,7 +20,7 @@ const TOOLS = [ { name: "search_docs", description: - "Search the Redis documentation and return the most relevant pages as references (id, title, url, summary, matching section ids). Returns no full text — follow up with get_page to read a result.", + "Search the Redis documentation and return the most relevant pages as references (title, url, summary, matching section ids). Each hit includes a unique `url` — pass that `url` to get_page (a hit's `id` is NOT unique and may be ambiguous). Returns no full text — follow up with get_page to read a result.", inputSchema: { type: "object" as const, properties: { @@ -44,12 +45,18 @@ const TOOLS = [ { name: "get_page", description: - "Fetch a single documentation page by 'id' or 'url'. Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", + "Fetch a single documentation page. Prefer the unique `url` from a search_docs hit. `id` also works but is NOT unique, so an ambiguous id returns an error listing candidate urls (likewise an ambiguous partial url). Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", inputSchema: { type: "object" as const, properties: { - id: { type: "string", description: "Page id (URL slug), e.g. 'commands/xadd'" }, - url: { type: "string", description: "Full or partial page URL" }, + id: { + type: "string", + description: "Page id (last URL slug); not unique — prefer url", + }, + url: { + type: "string", + description: "Full page URL (preferred — unique). A partial/suffix URL also works when it matches exactly one page.", + }, roles: { type: "array", items: { type: "string" }, @@ -60,17 +67,6 @@ const TOOLS = [ }, ]; -function ok(data: unknown) { - return { content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }] }; -} - -function fail(message: string) { - return { - content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], - isError: true, - }; -} - async function main() { const pages = await loadFeed(FEED_SOURCE); const index = new DocsIndex(pages); @@ -89,9 +85,9 @@ async function main() { try { switch (name) { case "search_docs": - return ok(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + return toolResult(searchDocs(index, SearchDocsInput.parse(args ?? {}))); case "get_page": - return ok(getPage(index, GetPageInput.parse(args ?? {}))); + return toolResult(getPage(index, GetPageInput.parse(args ?? {}))); default: return fail(`Unknown tool: ${name}`); } diff --git a/build/docs-mcp-server/node/src/response.ts b/build/docs-mcp-server/node/src/response.ts new file mode 100644 index 0000000000..6f1db590f1 --- /dev/null +++ b/build/docs-mcp-server/node/src/response.ts @@ -0,0 +1,24 @@ +// MCP tool-response helpers, factored out of index.ts so tests can import them +// without triggering index.ts's main() (which starts the stdio server). + +/** + * Serialise a tool result. If the tool returned an object carrying an `error` + * field (e.g. get_page couldn't resolve the page, or an id/url was ambiguous), + * mark the MCP response as an error so clients don't treat a failed lookup as + * success. + */ +export function toolResult(data: unknown) { + const isError = typeof data === "object" && data !== null && "error" in data; + return { + content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }], + ...(isError ? { isError: true } : {}), + }; +} + +/** For protocol-level failures (unknown tool, input parse errors). */ +export function fail(message: string) { + return { + content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], + isError: true, + }; +} diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index d177d0556a..ca2804e0e8 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -86,7 +86,10 @@ export interface SearchHit { export class DocsIndex { readonly pages: Page[]; - private byId = new Map(); + // The feed's `id` is the last URL path segment (e.g. "config", "acl") and is + // NOT unique — ~200 ids map to several pages. So id -> list, and callers must + // disambiguate by url. `url` IS unique, so byUrl stays 1:1. + private byId = new Map(); private byUrl = new Map(); private docs: Array<{ page: Page; @@ -104,7 +107,9 @@ export class DocsIndex { this.pages = pages; let totalLen = 0; for (const p of pages) { - this.byId.set(p.id, p); + const bucket = this.byId.get(p.id); + if (bucket) bucket.push(p); + else this.byId.set(p.id, [p]); if (p.url) this.byUrl.set(normalizeUrl(p.url), p); const tokens = tokenize(searchableText(p)); @@ -130,22 +135,25 @@ export class DocsIndex { return this.pages.length; } - getById(id: string): Page | undefined { - return this.byId.get(id); + /** All pages sharing this id (usually one, but the feed's id is not unique). */ + getPagesById(id: string): Page[] { + return this.byId.get(id) ?? []; } getByUrl(url: string): Page | undefined { return this.byUrl.get(normalizeUrl(url)); } - /** Fallback lookup when a caller passes a path or partial URL. */ - findByUrlSuffix(url: string): Page | undefined { + /** + * Fallback lookup when a caller passes a path or partial URL. Returns EVERY + * page whose url ends with the given suffix — a suffix can match several + * pages (e.g. "/install/"), so the caller must disambiguate rather than + * silently taking the first. + */ + matchByUrlSuffix(url: string): Page[] { const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); - if (!target) return undefined; - for (const p of this.pages) { - if (normalizeUrl(p.url).endsWith(target)) return p; - } - return undefined; + if (!target) return []; + return this.pages.filter((p) => normalizeUrl(p.url).endsWith(target)); } search(query: string, opts: SearchOptions = {}): SearchHit[] { diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index feef8ac118..4a3cbb30f0 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -1,11 +1,21 @@ -// Offline smoke test: exercises the index + tools directly (no MCP transport). -// npm run smoke # uses test/fixture.ndjson -// DOCS_NDJSON=... npm run smoke # point at a real feed (path or URL) +// Offline smoke test: assertion-based checks of the index + tools + MCP +// response wrapping (no stdio transport). Exits non-zero on any failure. +// npm run smoke +// Runs against test/fixture.ndjson; the assertions encode fixture-specific +// ids/urls (incl. the colliding id="install" pair), so it is not meant to be +// pointed at the live feed. import { fileURLToPath } from "node:url"; import { loadFeed } from "./feed.js"; import { DocsIndex } from "./search.js"; import { searchDocs } from "./tools/search-docs.js"; import { getPage } from "./tools/get-page.js"; +import { toolResult } from "./response.js"; + +let failures = 0; +function check(label: string, cond: boolean) { + console.log(`${cond ? "PASS" : "FAIL"} ${label}`); + if (!cond) failures++; +} const feed = process.env.DOCS_NDJSON ?? @@ -15,14 +25,38 @@ const pages = await loadFeed(feed); const index = new DocsIndex(pages); console.log(`loaded ${index.size} pages from ${feed}\n`); -console.log("# search_docs('append an entry to a stream')"); -console.log(JSON.stringify(searchDocs(index, { query: "append an entry to a stream" }), null, 2)); +// --- search_docs --- +const stream = searchDocs(index, { query: "append an entry to a stream" }); +check("search returns hits", stream.count > 0); +check("search hits carry a url", Boolean(stream.results[0]?.url)); + +// --- get_page happy paths --- +const xadd = getPage(index, { id: "commands/xadd", roles: ["parameters"] }) as any; +check("get_page(unique id) resolves", xadd.id === "commands/xadd"); +check("roles filter returns only 'parameters'", (xadd.sections ?? []).every((s: any) => s.role === "parameters")); + +const exact = getPage(index, { url: "https://redis.io/docs/latest/operate/redisinsight/install/" }) as any; +check("get_page(exact url) resolves the right page", exact.title === "Install Redis Insight"); + +const suffixUnique = getPage(index, { url: "/commands/xadd/" }) as any; +check("get_page(unambiguous partial url) resolves", suffixUnique.id === "commands/xadd"); + +// --- get_page ambiguity (Bugbot High + Codex Medium) --- +const ambId = getPage(index, { id: "install" }) as any; +check("ambiguous id returns error", typeof ambId.error === "string"); +check("ambiguous id lists candidates", (ambId.candidates ?? []).length === 2); + +const ambUrl = getPage(index, { url: "/install/" }) as any; +check("ambiguous partial url returns error (not silent first match)", typeof ambUrl.error === "string"); +check("ambiguous partial url lists candidates", (ambUrl.candidates ?? []).length === 2); -console.log("\n# search_docs('store json document', page_type=content)"); -console.log(JSON.stringify(searchDocs(index, { query: "store json document", page_type: "content" }), null, 2)); +const missing = getPage(index, { id: "does-not-exist-anywhere" }) as any; +check("missing page returns error", typeof missing.error === "string"); -console.log("\n# get_page(id='commands/xadd', roles=['parameters'])"); -console.log(JSON.stringify(getPage(index, { id: "commands/xadd", roles: ["parameters"] }), null, 2)); +// --- MCP response wrapping (Fix 2 / Bugbot Medium) --- +check("toolResult(missing) sets isError", toolResult(missing).isError === true); +check("toolResult(ambiguous id) sets isError", toolResult(ambId).isError === true); +check("toolResult(search) does NOT set isError", toolResult(stream).isError === undefined); -console.log("\n# get_page(url='/commands/xadd/') -- partial-URL fallback"); -console.log(JSON.stringify(getPage(index, { url: "/commands/xadd/", roles: ["return"] }).url ?? "not found", null, 2)); +console.log(`\n${failures === 0 ? "ALL PASSED" : failures + " FAILED"}`); +if (failures > 0) process.exit(1); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index 96b5fbba59..107cc486a3 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -1,5 +1,6 @@ import { z } from "zod"; import type { DocsIndex } from "../search.js"; +import type { Page } from "../types.js"; export const GetPageInput = z .object({ @@ -12,14 +13,44 @@ export const GetPageInput = z }); export type GetPageInput = z.infer; +/** An ambiguous id/url lookup: report all candidates so the caller can retry. */ +function ambiguous(kind: "id" | "url", value: string, matches: Page[]) { + return { + error: `Ambiguous ${kind} '${value}' matches ${matches.length} pages. Call get_page again with a full, exact url.`, + candidates: matches.map((p) => ({ title: p.title, url: p.url })), + }; +} + /** Fetch one page, optionally filtered to sections with the given roles. */ export function getPage(index: DocsIndex, input: GetPageInput) { - let page = input.id ? index.getById(input.id) : undefined; - if (!page && input.url) { - page = index.getByUrl(input.url) ?? index.findByUrlSuffix(input.url); + // Prefer url: a full url is unique. A partial/suffix url can match several + // pages, so disambiguate rather than silently taking the first. + let page: Page | undefined; + if (input.url) { + page = index.getByUrl(input.url); + if (!page) { + const matches = index.matchByUrlSuffix(input.url); + if (matches.length === 1) { + page = matches[0]; + } else if (matches.length > 1) { + return ambiguous("url", input.url, matches); + } + } } + + // id is the last URL path segment and can match several pages, so resolve by + // id only when unambiguous. + if (!page && input.id) { + const matches = index.getPagesById(input.id); + if (matches.length === 1) { + page = matches[0]; + } else if (matches.length > 1) { + return ambiguous("id", input.id, matches); + } + } + if (!page) { - return { error: `Page not found for ${JSON.stringify(input.id ?? input.url)}` }; + return { error: `Page not found for ${JSON.stringify(input.url ?? input.id)}` }; } let sections = page.sections ?? []; diff --git a/build/docs-mcp-server/node/test/fixture.ndjson b/build/docs-mcp-server/node/test/fixture.ndjson index 6e0c4ca8b8..95b767477c 100644 --- a/build/docs-mcp-server/node/test/fixture.ndjson +++ b/build/docs-mcp-server/node/test/fixture.ndjson @@ -1,3 +1,5 @@ {"id":"commands/xadd","title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/","summary":"Appends a new entry to a stream.","page_type":"content","content_hash":"abc123","sections":[{"id":"overview","title":"XADD","role":"overview","text":"Appends the specified stream entry to the stream at the specified key. If the key does not exist, a new stream is created."},{"id":"parameters","title":"Parameters","role":"parameters","text":"key: the name of the stream. NOMKSTREAM: optional, do not create the stream if it does not exist. field value: one or more field-value pairs to add to the stream entry."},{"id":"return","title":"Return value","role":"returns","text":"Returns the ID of the added stream entry."}],"examples":[{"id":"overview-ex0","language":"python","code":"r.xadd('mystream', {'field': 'value'})","section_id":"overview"}],"children":[]} {"id":"develop/data-types/json","title":"JSON","url":"https://redis.io/docs/latest/develop/data-types/json/","summary":"Store and query JSON documents in Redis.","page_type":"content","content_hash":"def456","sections":[{"id":"overview","title":"Redis JSON","role":"overview","text":"The JSON data type lets you store, update, and retrieve JSON values in a Redis database."},{"id":"example","title":"Example","role":"example","text":"Use JSON.SET to store a document and JSON.GET to retrieve values from it by path."}],"examples":[{"id":"example-ex0","language":"python","code":"r.json().set('doc', '$', {'a': 1})","section_id":"example"}],"children":[]} {"id":"commands","title":"Commands","url":"https://redis.io/docs/latest/commands/","summary":"Reference documentation for all Redis commands.","page_type":"index","children":[{"title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/"}]} +{"id":"install","title":"Install Redis","url":"https://redis.io/docs/latest/operate/oss_and_stack/install/","summary":"Install Redis Open Source on your platform.","page_type":"content","content_hash":"aaa111","sections":[{"id":"overview","title":"Install Redis","role":"overview","text":"Install Redis Open Source on Linux, macOS, or Windows."}],"examples":[],"children":[]} +{"id":"install","title":"Install Redis Insight","url":"https://redis.io/docs/latest/operate/redisinsight/install/","summary":"Install the Redis Insight GUI.","page_type":"content","content_hash":"bbb222","sections":[{"id":"overview","title":"Install Redis Insight","role":"overview","text":"Download and install the Redis Insight desktop application."}],"examples":[],"children":[]} From 97b2dd29d2956b8969807cdf671072e03c52564c Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:09:48 +0100 Subject: [PATCH 3/9] DOC-6809 Add local MCP client harness; note role-vocabulary gap in spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test/mcp-client.mjs drives the built stdio server through the real MCP client (spawn -> initialize -> tools/list -> tools/call) against the fixture, so the transport path and isError signalling are exercised end-to-end, not just the tool logic. Live-feed testing surfaced that the spec's assumed section roles (syntax/examples) don't match the feed (it uses content/parameters/example/ returns, no syntax), so a roles filter can silently return nothing — recorded in SPEC open questions to resolve before building get_examples/get_command. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 9 ++++ .../docs-mcp-server/node/test/mcp-client.mjs | 44 +++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 build/docs-mcp-server/node/test/mcp-client.mjs diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 3a8a739d21..cdc8c1dba3 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -202,6 +202,15 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. overviews** that repeat the same terms. Fix needs an analyzer (stemming/lemmatization), possibly a page-type/canonical boost, and/or the vector-search path. This is the strongest argument for §6's v2 upgrade. +- **Section-role vocabulary (found via live MCP test):** the roles the spec + assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match + the feed. Command pages actually carry `content` / `parameters` / `example` + (singular) / `returns` — there is no `syntax` role, and it's `example` not + `examples`. So a `roles: ["examples","syntax"]` filter returns **zero + sections** against the real feed (verified on EXPIREAT). Before building + `get_examples` / `get_command` and documenting `roles`, enumerate the actual + role set across the corpus and align tool params/docs to it (and decide + whether the server should normalise synonyms like `examples`→`example`). - **`get_command` coverage:** command pages *do* carry `parameters`/`returns`/ `example` roles (confirmed: `get_page('expire')` → roles `[content, parameters, example, returns]`). Confirm this holds across all diff --git a/build/docs-mcp-server/node/test/mcp-client.mjs b/build/docs-mcp-server/node/test/mcp-client.mjs new file mode 100644 index 0000000000..5010723f0e --- /dev/null +++ b/build/docs-mcp-server/node/test/mcp-client.mjs @@ -0,0 +1,44 @@ +// Manual local test: drive the built stdio server through the real MCP client +// (spawn -> initialize -> tools/list -> tools/call), against the local fixture. +// node test/mcp-client.mjs +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"; +import { fileURLToPath } from "node:url"; + +const feed = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./fixture.ndjson", import.meta.url)); +const serverEntry = fileURLToPath(new URL("../dist/index.js", import.meta.url)); + +const transport = new StdioClientTransport({ + command: "node", + args: [serverEntry], + env: { ...process.env, DOCS_NDJSON: feed }, +}); + +const client = new Client({ name: "local-test-client", version: "0.0.1" }, { capabilities: {} }); +await client.connect(transport); +console.log("connected to server\n"); + +const tools = await client.listTools(); +console.log("tools/list ->", tools.tools.map((t) => t.name).join(", "), "\n"); + +const search = await client.callTool({ + name: "search_docs", + arguments: { query: "append an entry to a stream" }, +}); +console.log("tools/call search_docs('append an entry to a stream'):"); +console.log(search.content[0].text, "\n"); + +const amb = await client.callTool({ name: "get_page", arguments: { id: "install" } }); +console.log(`tools/call get_page(id='install') -> isError=${amb.isError}`); +console.log(amb.content[0].text, "\n"); + +const page = await client.callTool({ + name: "get_page", + arguments: { url: "https://redis.io/docs/latest/commands/xadd/", roles: ["parameters"] }, +}); +console.log(`tools/call get_page(url=.../commands/xadd/, roles=['parameters']) -> isError=${page.isError}`); +console.log(page.content[0].text); + +await client.close(); +console.log("\nclosed cleanly"); From e94b8562ee74eff2b2cec4307474654fec96be24 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:30:22 +0100 Subject: [PATCH 4/9] DOC-6809 Consolidate get_page resolution; drop unbacked version param MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-2 Bugbot findings on the get_page/search resolution logic were churn: the round-1 duplicate-id fix introduced two of them. Rather than three point- patches, consolidate resolution into one convergence model (per Codex design review): collect candidate pages across all supplied handles (exact url, id, boundary-anchored suffix url) and resolve only when they converge on exactly one page — so an ambiguous url still tries id, and url/id pointing at different pages reports conflicting handles instead of silently applying precedence. - Partial-url suffix matching is now anchored to "/" path segments, so "get" matches .../commands/get but not .../config-get or .../arget (was 19 false matches on the live feed, now 2 legitimate ones). [Bugbot High] - Ambiguous url no longer short-circuits the id fallback. [Bugbot Medium] - Removed the `version` param: it advertised a `latest` default it never enforced. Deferred until a committed URL/version model exists. [Bugbot Medium] - Smoke test extended with boundary and conflicting-handles assertions (17 total). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 13 +++- build/docs-mcp-server/node/README.md | 5 +- build/docs-mcp-server/node/src/index.ts | 4 -- build/docs-mcp-server/node/src/search.ts | 26 +++---- build/docs-mcp-server/node/src/smoke.ts | 12 ++++ .../node/src/tools/get-page.ts | 71 +++++++++++-------- .../node/src/tools/search-docs.ts | 2 - 7 files changed, 74 insertions(+), 59 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index cdc8c1dba3..3bbd70272d 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -170,9 +170,16 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. ## 7. Versioning -- `search_docs` / `get_page` accept an optional `version`; default to `latest`. -- Responses echo the resolved version so an agent can't silently blend - versions. This is the single most common RAG-over-docs correctness bug. +- **Planned:** `search_docs` / `get_page` accept an optional `version` (default + `latest`), and responses echo the resolved version so an agent can't silently + blend versions — the single most common RAG-over-docs correctness bug. +- **v0 status: deferred, not implemented.** The prototype does **not** expose a + `version` param. An earlier v0 advertised a `latest` default it didn't + enforce (the filter was a no-op, and the live feed is single-version anyway), + which misleads agents. Rather than bake in a `page.url.includes("//")` + heuristic, the param was removed until a committed URL/version model exists + (Bugbot #3585 + Codex review). Add it back with the real filter when the feed + carries multiple version trees. ## 8. Freshness diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 0bd4af5209..3995ca0121 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -51,7 +51,7 @@ Add to a Claude Code / Cursor MCP config after `npm run build`: | Tool | Inputs | Returns | |------|--------|---------| -| `search_docs` | `query` (req), `page_type`, `version`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | +| `search_docs` | `query` (req), `page_type`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | | `get_page` | `id` **or** `url` (req), `roles[]` | one page with `content_hash` + `sections`, optionally filtered to the given section roles | Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` @@ -73,6 +73,7 @@ Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` terms. Fixing this properly means an analyzer (stemming/lemmatization) and/or the vector-search upgrade tracked in SPEC §6/§10. -- **Version filtering is heuristic** (URL-path based); see SPEC §7. +- **No version filtering.** The `version` param was removed until a committed + URL/version model exists (the feed is single-version today); see SPEC §7. - Only `search_docs` + `get_page`. `get_examples`, `get_section`, `get_command` are v1 (SPEC §4/§11). diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts index 43ac439c42..21fbcd51ec 100644 --- a/build/docs-mcp-server/node/src/index.ts +++ b/build/docs-mcp-server/node/src/index.ts @@ -30,10 +30,6 @@ const TOOLS = [ enum: ["content", "index"], description: "Restrict to prose ('content') or navigation ('index') pages", }, - version: { - type: "string", - description: "Redis docs version, e.g. 'latest' (default) or '7.4'", - }, limit: { type: "number", description: "Max results (default 10, max 50)", diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index ca2804e0e8..9e54cd0efb 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -57,21 +57,9 @@ function matchingSections(p: Page, qterms: Set): string[] { return out; } -/** - * Heuristic version filter. Redis docs carry the version in the URL path - * (e.g. /docs/latest/... or /docs/7.4/...). "latest"/unset does not filter. - * NOTE: approximate pending confirmation of how versions appear in the feed - * (SPEC.md §7 / open question). - */ -function matchesVersion(page: Page, version: string): boolean { - if (!version || version.toLowerCase() === "latest") return true; - return page.url.includes(`/${version}/`); -} - export interface SearchOptions { limit?: number; pageType?: string; - version?: string; } export interface SearchHit { @@ -146,14 +134,19 @@ export class DocsIndex { /** * Fallback lookup when a caller passes a path or partial URL. Returns EVERY - * page whose url ends with the given suffix — a suffix can match several - * pages (e.g. "/install/"), so the caller must disambiguate rather than - * silently taking the first. + * page whose url ends with the given suffix **at a path-segment boundary**, + * so "get" matches ".../commands/get" but NOT ".../config-get" or + * ".../arget". A suffix can still match several pages (e.g. "/install/" or a + * bare last segment shared by many pages), so the caller must disambiguate. */ matchByUrlSuffix(url: string): Page[] { const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); if (!target) return []; - return this.pages.filter((p) => normalizeUrl(p.url).endsWith(target)); + // Anchor the leading edge to a "/" so we match whole path segments. The + // trailing edge is already anchored: normalizeUrl strips the trailing slash + // and we compare against the end of the string. + const anchored = target.startsWith("/") ? target : `/${target}`; + return this.pages.filter((p) => normalizeUrl(p.url).endsWith(anchored)); } search(query: string, opts: SearchOptions = {}): SearchHit[] { @@ -173,7 +166,6 @@ export class DocsIndex { const hits: SearchHit[] = []; for (const d of this.docs) { if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; - if (opts.version && !matchesVersion(d.page, opts.version)) continue; let score = 0; for (const t of qterms) { diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index 4a3cbb30f0..a44475d6cd 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -50,6 +50,18 @@ const ambUrl = getPage(index, { url: "/install/" }) as any; check("ambiguous partial url returns error (not silent first match)", typeof ambUrl.error === "string"); check("ambiguous partial url lists candidates", (ambUrl.candidates ?? []).length === 2); +// --- boundary-anchored suffix (Bugbot round-2 High): "add" must NOT match "xadd" --- +const boundary = getPage(index, { url: "add" }) as any; +check("partial url matches only on path-segment boundary (add !-> xadd)", typeof boundary.error === "string" && !boundary.candidates); + +// --- conflicting handles (Codex convergence model): url and id point at different pages --- +const conflict = getPage(index, { + url: "https://redis.io/docs/latest/develop/data-types/json/", + id: "commands/xadd", +}) as any; +check("conflicting url+id returns error", typeof conflict.error === "string"); +check("conflicting url+id lists both candidates", (conflict.candidates ?? []).length === 2); + const missing = getPage(index, { id: "does-not-exist-anywhere" }) as any; check("missing page returns error", typeof missing.error === "string"); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index 107cc486a3..cbe9bd7f25 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -13,46 +13,55 @@ export const GetPageInput = z }); export type GetPageInput = z.infer; -/** An ambiguous id/url lookup: report all candidates so the caller can retry. */ -function ambiguous(kind: "id" | "url", value: string, matches: Page[]) { - return { - error: `Ambiguous ${kind} '${value}' matches ${matches.length} pages. Call get_page again with a full, exact url.`, - candidates: matches.map((p) => ({ title: p.title, url: p.url })), +/** + * The distinct pages (deduped by unique url) that the supplied handles resolve + * to. `url` is authoritative when it matches an exact page; otherwise it is + * treated as a path-boundary suffix. `id` is the last URL segment and is not + * unique, so it may contribute several pages. Collecting across all handles + * (rather than short-circuiting) means an ambiguous url still lets `id` help, + * and it surfaces the case where url and id point at *different* pages. + */ +function collectCandidates(index: DocsIndex, input: GetPageInput): Page[] { + const byUrl = new Map(); + const add = (p: Page | undefined) => { + if (p) byUrl.set(p.url, p); }; + + if (input.url) { + const exact = index.getByUrl(input.url); + if (exact) add(exact); + else index.matchByUrlSuffix(input.url).forEach(add); + } + if (input.id) { + index.getPagesById(input.id).forEach(add); + } + return [...byUrl.values()]; +} + +function describeHandles(input: GetPageInput): string { + const parts: string[] = []; + if (input.url) parts.push(`url '${input.url}'`); + if (input.id) parts.push(`id '${input.id}'`); + return parts.join(" and "); } /** Fetch one page, optionally filtered to sections with the given roles. */ export function getPage(index: DocsIndex, input: GetPageInput) { - // Prefer url: a full url is unique. A partial/suffix url can match several - // pages, so disambiguate rather than silently taking the first. - let page: Page | undefined; - if (input.url) { - page = index.getByUrl(input.url); - if (!page) { - const matches = index.matchByUrlSuffix(input.url); - if (matches.length === 1) { - page = matches[0]; - } else if (matches.length > 1) { - return ambiguous("url", input.url, matches); - } - } - } + const candidates = collectCandidates(index, input); - // id is the last URL path segment and can match several pages, so resolve by - // id only when unambiguous. - if (!page && input.id) { - const matches = index.getPagesById(input.id); - if (matches.length === 1) { - page = matches[0]; - } else if (matches.length > 1) { - return ambiguous("id", input.id, matches); - } + if (candidates.length === 0) { + return { error: `Page not found for ${describeHandles(input)}.` }; } - - if (!page) { - return { error: `Page not found for ${JSON.stringify(input.url ?? input.id)}` }; + if (candidates.length > 1) { + // Ambiguous (a non-unique id / boundary suffix) or conflicting (url and id + // point at different pages) — either way, make the caller pick a url. + return { + error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, + candidates: candidates.map((p) => ({ title: p.title, url: p.url })), + }; } + const page = candidates[0]; let sections = page.sections ?? []; if (input.roles && input.roles.length) { const want = new Set(input.roles.map((r) => r.toLowerCase())); diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts index 32313207ed..08f1574feb 100644 --- a/build/docs-mcp-server/node/src/tools/search-docs.ts +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -4,7 +4,6 @@ import type { DocsIndex } from "../search.js"; export const SearchDocsInput = z.object({ query: z.string().min(1, "query is required"), page_type: z.enum(["content", "index"]).optional(), - version: z.string().optional(), limit: z.number().int().positive().max(50).optional(), }); export type SearchDocsInput = z.infer; @@ -14,7 +13,6 @@ export function searchDocs(index: DocsIndex, input: SearchDocsInput) { const results = index.search(input.query, { limit: input.limit ?? 10, pageType: input.page_type, - version: input.version, }); return { query: input.query, count: results.length, results }; } From 7f8ef54f67d64853557d3ab1e0e10a56512868ba Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:38:56 +0100 Subject: [PATCH 5/9] DOC-6809 Add retrieval eval harness; baseline shows lexical is insufficient MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit npm run eval scores search_docs retrieval against a curated question set (test/eval/cases.json — command lookups phrased without the command name), reporting recall@k / MRR with a data-integrity check that flags expected urls missing from the feed. Feed read from DOCS_NDJSON or a gitignored local cache. Baseline on the live feed (22 cases): recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42. Confirms with data what was hypothesised: lexical retrieval alone is not good enough — canonical command pages lose to sibling commands (zadd -> zincrby), operator pages (del -> remove-node), and concept pages (ft.create -> a full-text concept page), and there is no stemming (append !-> appends). This is the measured case for the stemming/boost/vector work in SPEC §6/§10 and lets future ranking changes be compared against a baseline instead of tuned blind. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/.gitignore | 1 + build/docs-mcp-server/node/README.md | 17 ++++- build/docs-mcp-server/node/package.json | 3 +- .../docs-mcp-server/node/test/eval/cases.json | 24 +++++++ build/docs-mcp-server/node/test/eval/run.mjs | 72 +++++++++++++++++++ 5 files changed, 115 insertions(+), 2 deletions(-) create mode 100644 build/docs-mcp-server/node/test/eval/cases.json create mode 100644 build/docs-mcp-server/node/test/eval/run.mjs diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore index 3c25e1e49c..d458242e86 100644 --- a/build/docs-mcp-server/node/.gitignore +++ b/build/docs-mcp-server/node/.gitignore @@ -1,3 +1,4 @@ node_modules/ dist/ *.log +test/eval/docs.ndjson* diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 3995ca0121..f85e4977e7 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -57,11 +57,26 @@ Add to a Claude Code / Cursor MCP config after `npm run build`: Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` (e.g. `["parameters","returns"]`) to pull just what's needed. -## Measured (real feed, 2,531 pages) +## Measured (real feed, ~2,530 pages) - Index build: ~0.3 s. Query latency: ~75–125 ms. This is why v0 needs no datastore and why Rust/WASM would be premature (see SPEC §6). +## Retrieval eval + +`npm run eval` scores retrieval quality: it runs the questions in +`test/eval/cases.json` (command-lookup questions phrased *without* the command +name) through `search_docs` and reports recall@k / MRR, plus a data-integrity +check that flags any expected url missing from the feed. The feed is read from +`DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; +`curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). + +**Baseline (lexical v0):** recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42 — i.e. +lexical retrieval is **not** good enough alone (canonical command pages lose to +sibling commands, operator, and concept pages; no stemming). This is the +measured case for the ranking / vector-search work in SPEC §6/§10. Use it to +compare any ranking change against the baseline rather than tuning blind. + ## Known limitations (v0) - **Ranking is untuned lexical search.** Good on distinctive terms diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json index ce615096b0..90f51e6444 100644 --- a/build/docs-mcp-server/node/package.json +++ b/build/docs-mcp-server/node/package.json @@ -11,7 +11,8 @@ "build": "tsc", "start": "tsx src/index.ts", "dev": "tsx watch src/index.ts", - "smoke": "tsx src/smoke.ts" + "smoke": "tsx src/smoke.ts", + "eval": "npm run build && node test/eval/run.mjs" }, "keywords": [ "redis", diff --git a/build/docs-mcp-server/node/test/eval/cases.json b/build/docs-mcp-server/node/test/eval/cases.json new file mode 100644 index 0000000000..382331819e --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/cases.json @@ -0,0 +1,24 @@ +[ + { "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, + { "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, + { "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, + { "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, + { "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, + { "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, + { "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, + { "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, + { "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, + { "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, + { "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, + { "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, + { "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, + { "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, + { "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, + { "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, + { "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, + { "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, + { "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, + { "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, + { "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, + { "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] } +] diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs new file mode 100644 index 0000000000..c73964d8ca --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -0,0 +1,72 @@ +// AI-answer eval (retrieval quality) for the docs MCP server. +// Runs each question in cases.json through search_docs and measures whether the +// expected canonical page is retrieved (recall@k, MRR). A data-integrity check +// flags any expected url that isn't in the feed, so a bad ground-truth entry is +// reported rather than silently scored as a miss. +// +// node test/eval/run.mjs # uses cached test/eval/docs.ndjson.gz +// DOCS_NDJSON= node test/eval/run.mjs +// +// Imports the BUILT server (dist/), so run `npm run build` first. +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { loadFeed } from "../../dist/feed.js"; +import { DocsIndex } from "../../dist/search.js"; +import { searchDocs } from "../../dist/tools/search-docs.js"; + +const K = [1, 3, 5, 10]; +const LIMIT = 10; +const norm = (u) => u.trim().toLowerCase().replace(/\/+$/, ""); +const short = (u) => (u ? u.replace("https://redis.io/docs/latest", "") : "—"); + +const feedSrc = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./docs.ndjson.gz", import.meta.url)); +const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", import.meta.url)), "utf8")); + +const pages = await loadFeed(feedSrc); +const index = new DocsIndex(pages); +const feedUrls = new Set(pages.map((p) => norm(p.url))); + +const broken = []; +const rows = []; +for (const c of cases) { + const expected = c.expected.map(norm); + if (expected.every((u) => !feedUrls.has(u))) { + broken.push({ q: c.q, missing: expected }); + continue; + } + const results = searchDocs(index, { query: c.q, limit: LIMIT }).results.map((r) => norm(r.url)); + let rank = null; + for (let i = 0; i < results.length; i++) { + if (expected.includes(results[i])) { + rank = i + 1; + break; + } + } + rows.push({ q: c.q, rank, top: results[0] }); +} + +const scored = rows.length; +const recall = Object.fromEntries( + K.map((k) => [k, rows.filter((r) => r.rank && r.rank <= k).length / scored]), +); +const mrr = rows.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / scored; + +console.log( + `Feed: ${pages.length} pages | cases scored: ${scored}` + + (broken.length ? ` | ${broken.length} BROKEN (expected url not in feed)` : "") + + "\n", +); +for (const r of rows) { + const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; + console.log(`${tag} ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); +} + +console.log("\n--- retrieval quality ---"); +for (const k of K) console.log(`recall@${k}: ${(recall[k] * 100).toFixed(0)}%`); +console.log(`MRR: ${mrr.toFixed(3)}`); + +if (broken.length) { + console.log("\n--- BROKEN eval cases (fix ground truth) ---"); + for (const b of broken) console.log(` "${b.q}" -> not in feed: ${b.missing.map(short).join(", ")}`); +} From c5f4ebe296ea8447ca48ee06c01366526277a3e7 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 2 Jul 2026 16:45:48 +0100 Subject: [PATCH 6/9] DOC-6809 Add stemming + page-type weighting; recall@5 59%->86% MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two principled ranking changes, measured against the eval baseline: - Porter stemmer (src/stem.ts) applied to both index and query tokens so word forms conflate (append/appends, prepend/prepending, expires/expire). Stopwords are filtered on raw tokens before stemming. - Page-type weighting on the final score: demote release-notes/REST-API (x0.5) and operator (x0.7) pages, modestly boost /commands/* (x1.5) — targeting the observed failures where operator/concept pages outranked command pages. Eval (22 command-lookup cases): recall@1 32->50, @3 45->68, @5 59->86, @10 73->95, MRR 0.42->0.65; misses 6->1. Caveat recorded in README/SPEC: the eval is command-heavy so the /commands/* boost partly flatters it — concept/ how-to cases still needed before calling lexical sufficient generally, and the residual miss (del/unlink beaten by flushdb for "remove a key") is a semantic gap that motivates vector search. Smoke still green (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 20 ++-- build/docs-mcp-server/node/README.md | 33 +++--- build/docs-mcp-server/node/src/search.ts | 41 +++++-- build/docs-mcp-server/node/src/stem.ts | 136 +++++++++++++++++++++++ 4 files changed, 198 insertions(+), 32 deletions(-) create mode 100644 build/docs-mcp-server/node/src/stem.ts diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 3bbd70272d..97d26837b9 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -200,15 +200,17 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / RedisVL, and is the retrieval gain worth the added infra given how well-structured the corpus already is? -- **Ranking quality (found in v0 prototype):** untuned lexical BM25 — even - with stopword removal and title/summary/slug field boosts — mis-ranks - natural-language queries. Two concrete failure modes observed on the live - feed: (1) **no stemming**, so "append" ≠ "appends" and `XADD` won't surface - for "append an entry to a stream"; (2) **canonical command pages compete with - release notes, operator `custom-resources` pages, and client-library - overviews** that repeat the same terms. Fix needs an analyzer - (stemming/lemmatization), possibly a page-type/canonical boost, and/or the - vector-search path. This is the strongest argument for §6's v2 upgrade. +- **Ranking quality (measured via the eval harness):** lexical BM25 with + Porter stemming, stopword removal, title/summary/slug field boosts, and + page-type weighting (demote release-notes/REST-API/operator, modestly boost + `/commands/*`) gets **recall@5 86% / MRR 0.65** on 22 command-lookup cases — + up from a **59% / 0.42** un-stemmed, un-weighted baseline. Two caveats: (1) + the eval is command-heavy so the `/commands/*` boost partly flatters it — + **add concept/how-to eval cases** before concluding lexical is sufficient + generally; (2) the residual misses are pure semantic gaps ("remove a key" → + `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting + **weakened but did not eliminate** the §6 vector-search case — the eval now + lets that call be made on numbers. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index f85e4977e7..2cd83d5dc6 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -71,23 +71,28 @@ check that flags any expected url missing from the feed. The feed is read from `DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; `curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). -**Baseline (lexical v0):** recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42 — i.e. -lexical retrieval is **not** good enough alone (canonical command pages lose to -sibling commands, operator, and concept pages; no stemming). This is the -measured case for the ranking / vector-search work in SPEC §6/§10. Use it to -compare any ranking change against the baseline rather than tuning blind. +**Results (22 command-lookup cases):** + +| | recall@1 | recall@3 | recall@5 | recall@10 | MRR | +|---|---|---|---|---|---| +| lexical baseline | 32% | 45% | 59% | 73% | 0.42 | +| + Porter stemming + page-type weighting | 50% | 68% | 86% | 95% | 0.65 | + +Stemming (append↔appends) and demoting secondary pages (release-notes / REST-API / +operator) while modestly boosting `/commands/*` lifted recall@5 from 59% → 86%. +**Caveat:** this eval is command-heavy, so the `/commands/*` boost partly flatters +it — concept/how-to query cases are not yet covered and should be added before +concluding lexical is sufficient generally. The remaining case (`del`/`unlink` +for "remove a key", beaten by `flushdb`) is a genuine lexical limitation and is +the kind of gap vector search would close (SPEC §6/§10). ## Known limitations (v0) -- **Ranking is untuned lexical search.** Good on distinctive terms - (`publish`, `incrby`, `pexpire`), but weaker where: - - there is **no stemmer** — a query for "append" won't match a summary that - says "appends", so `XADD` is hard to surface from natural language; - - **canonical command pages** compete with release notes, operator - (custom-resources) pages, and client-library overviews that repeat the same - terms. - Fixing this properly means an analyzer (stemming/lemmatization) and/or the - vector-search upgrade tracked in SPEC §6/§10. +- **Ranking is lexical (BM25 + Porter stemming + field/page-type weighting).** + Now recall@5 86% on command lookups (see eval above), but still lexical: it + can't bridge pure semantic gaps (e.g. "remove a key" → `flushdb` over `del`), + and concept/how-to queries are unmeasured. Closing the remainder is the case + for vector search (SPEC §6/§10). - **No version filtering.** The `version` param was removed until a committed URL/version model exists (the feed is single-version today); see SPEC §7. - Only `search_docs` + `get_page`. `get_examples`, `get_section`, diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index 9e54cd0efb..d5d913285d 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -1,4 +1,5 @@ import type { Page } from "./types.js"; +import { stem } from "./stem.js"; // Self-contained BM25 lexical index. No external search dependency: at // ~4,100 docs the whole index builds in-memory in well under a second, which @@ -7,6 +8,20 @@ import type { Page } from "./types.js"; const K1 = 1.5; const B = 0.75; +// Page-type weighting applied to the final score. Command reference pages are +// the canonical answer for command-shaped queries; release-notes / REST-API / +// operator pages are secondary and were observed outranking primary docs +// (e.g. "remove a key" -> operate/.../remove-node). Multipliers, not filters. +// NOTE: the retrieval eval is command-heavy, so keep the command boost modest +// to avoid tuning to the eval — the demotions are the more general fix. +function pageWeight(url: string): number { + const u = url.toLowerCase(); + if (u.includes("/release-notes") || u.includes("/rest-api/")) return 0.5; + if (u.includes("/operate/")) return 0.7; + if (u.includes("/commands/")) return 1.5; + return 1; +} + // Field boosts (added on top of the body BM25 score, weighted by term idf). // A query term appearing in the title/slug/summary is a strong signal that the // page is *about* that term — this lifts canonical command pages (whose summary @@ -28,6 +43,12 @@ function tokenize(text: string): string[] { return text.toLowerCase().match(/[a-z0-9]+/g) ?? []; } +/** Tokenize + stem. Used for everything indexed and for query terms, so word + * forms conflate. Stopwords are filtered on RAW tokens before this (see search). */ +function analyze(text: string): string[] { + return tokenize(text).map(stem); +} + function normalizeUrl(u: string): string { return u.trim().toLowerCase().replace(/\/+$/, ""); } @@ -45,7 +66,7 @@ function searchableText(p: Page): string { function matchingSections(p: Page, qterms: Set): string[] { const out: string[] = []; for (const s of p.sections ?? []) { - const toks = new Set(tokenize(`${s.title ?? ""} ${s.text ?? ""}`)); + const toks = new Set(analyze(`${s.title ?? ""} ${s.text ?? ""}`)); for (const t of qterms) { if (toks.has(t)) { out.push(s.id); @@ -100,7 +121,7 @@ export class DocsIndex { else this.byId.set(p.id, [p]); if (p.url) this.byUrl.set(normalizeUrl(p.url), p); - const tokens = tokenize(searchableText(p)); + const tokens = analyze(searchableText(p)); if (tokens.length === 0) continue; const tf = new Map(); for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); @@ -109,9 +130,9 @@ export class DocsIndex { page: p, tf, len: tokens.length, - titleTok: new Set(tokenize(p.title ?? "")), - slugTok: new Set(tokenize(p.id ?? "")), - summaryTok: new Set(tokenize(p.summary ?? "")), + titleTok: new Set(analyze(p.title ?? "")), + slugTok: new Set(analyze(p.id ?? "")), + summaryTok: new Set(analyze(p.summary ?? "")), }); totalLen += tokens.length; } @@ -150,10 +171,11 @@ export class DocsIndex { } search(query: string, opts: SearchOptions = {}): SearchHit[] { - let qterms = [...new Set(tokenize(query))].filter((t) => !STOPWORDS.has(t)); - // If the query was *all* stopwords, fall back to using them rather than - // returning nothing. - if (qterms.length === 0) qterms = [...new Set(tokenize(query))]; + // Filter stopwords on RAW tokens (before stemming), then stem + dedupe. + const raw = [...new Set(tokenize(query))]; + let kept = raw.filter((t) => !STOPWORDS.has(t)); + if (kept.length === 0) kept = raw; // query was all stopwords + const qterms = [...new Set(kept.map(stem))]; if (qterms.length === 0) return []; const idf = new Map(); @@ -181,6 +203,7 @@ export class DocsIndex { if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; } if (score > 0) { + score *= pageWeight(d.page.url); hits.push({ id: d.page.id, title: d.page.title, diff --git a/build/docs-mcp-server/node/src/stem.ts b/build/docs-mcp-server/node/src/stem.ts new file mode 100644 index 0000000000..e94296f3b4 --- /dev/null +++ b/build/docs-mcp-server/node/src/stem.ts @@ -0,0 +1,136 @@ +// Compact Porter stemmer (classic algorithm). Applied to BOTH index and query +// tokens so word-form variants conflate (append/appends, prepend/prepending, +// expire/expires, queries/query). Consistency matters more than linguistic +// perfection here — the retrieval eval validates the net effect. + +const step2list: Record = { + ational: "ate", tional: "tion", enci: "ence", anci: "ance", izer: "ize", + bli: "ble", alli: "al", entli: "ent", eli: "e", ousli: "ous", + ization: "ize", ation: "ate", ator: "ate", alism: "al", iveness: "ive", + fulness: "ful", ousness: "ous", aliti: "al", iviti: "ive", biliti: "ble", + logi: "log", +}; +const step3list: Record = { + icate: "ic", ative: "", alize: "al", iciti: "ic", ical: "ic", ful: "", ness: "", +}; + +const c = "[^aeiou]"; +const v = "[aeiouy]"; +const C = c + "[^aeiouy]*"; +const V = v + "[aeiou]*"; +const mgr0 = "^(" + C + ")?" + V + C; +const meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; +const mgr1 = "^(" + C + ")?" + V + C + V + C; +const s_v = "^(" + C + ")?" + v; + +export function stem(w: string): string { + if (w.length < 3) return w; + + let stemmed: string; + let suffix: string; + let re: RegExp; + let re2: RegExp; + let re3: RegExp; + let re4: RegExp; + + const firstch = w.substr(0, 1); + if (firstch === "y") w = firstch.toUpperCase() + w.substr(1); + + // Step 1a + re = /^(.+?)(ss|i)es$/; + re2 = /^(.+?)([^s])s$/; + if (re.test(w)) w = w.replace(re, "$1$2"); + else if (re2.test(w)) w = w.replace(re2, "$1$2"); + + // Step 1b + re = /^(.+?)eed$/; + re2 = /^(.+?)(ed|ing)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + re = new RegExp(mgr0); + if (re.test(fp[1])) { + re = /.$/; + w = w.replace(re, ""); + } + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1]; + re2 = new RegExp(s_v); + if (re2.test(stemmed)) { + w = stemmed; + re2 = /(at|bl|iz)$/; + re3 = new RegExp("([^aeiouylsz])\\1$"); + re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re2.test(w)) w = w + "e"; + else if (re3.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } else if (re4.test(w)) w = w + "e"; + } + } + + // Step 1c + re = /^(.+?)y$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(s_v); + if (re.test(stemmed)) w = stemmed + "i"; + } + + // Step 2 + re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step2list[suffix]; + } + + // Step 3 + re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step3list[suffix]; + } + + // Step 4 + re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; + re2 = /^(.+?)(s|t)(ion)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + if (re.test(stemmed)) w = stemmed; + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1] + fp[2]; + re2 = new RegExp(mgr1); + if (re2.test(stemmed)) w = stemmed; + } + + // Step 5a + re = /^(.+?)e$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + re2 = new RegExp(meq1); + re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re.test(stemmed) || (re2.test(stemmed) && !re3.test(stemmed))) w = stemmed; + } + + // Step 5b + re = /ll$/; + re2 = new RegExp(mgr1); + if (re.test(w) && re2.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } + + return w.toLowerCase(); +} From eb0e7ca72034021613a55a94f6f2e43b01ee37d8 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 10:47:12 +0100 Subject: [PATCH 7/9] DOC-6809 Make exact url authoritative in get_page (Bugbot round 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bugbot High: collectCandidates always merged id matches even when url already resolved to an exact page, so get_page(url=, id=) — the natural flow when an agent echoes both fields from a search hit — returned an ambiguous error instead of the page. The consolidation's "always converge" rule over-corrected: an exact url is unique and should win. Fix: an exact url short-circuits to that page, EXCEPT when a supplied id points somewhere else entirely (idPages don't include the exact page) — still a genuine conflict, so still reported. This reconciles the Bugbot finding with Codex's earlier conflict-detection ask: exact-url + its-own-id resolves; exact-url + a different id conflicts. Smoke covers both (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/node/src/smoke.ts | 8 +++ .../node/src/tools/get-page.ts | 69 ++++++++++++------- 2 files changed, 52 insertions(+), 25 deletions(-) diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts index a44475d6cd..fedc6c450d 100644 --- a/build/docs-mcp-server/node/src/smoke.ts +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -38,6 +38,14 @@ check("roles filter returns only 'parameters'", (xadd.sections ?? []).every((s: const exact = getPage(index, { url: "https://redis.io/docs/latest/operate/redisinsight/install/" }) as any; check("get_page(exact url) resolves the right page", exact.title === "Install Redis Insight"); +// exact url is authoritative even when a non-unique id is passed alongside it +// (Bugbot round-3 High): search hits carry both id + url, and id "install" is ambiguous. +const exactPlusId = getPage(index, { + url: "https://redis.io/docs/latest/operate/redisinsight/install/", + id: "install", +}) as any; +check("exact url + non-unique id resolves (not ambiguous)", exactPlusId.title === "Install Redis Insight"); + const suffixUnique = getPage(index, { url: "/commands/xadd/" }) as any; check("get_page(unambiguous partial url) resolves", suffixUnique.id === "commands/xadd"); diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts index cbe9bd7f25..d3326423b8 100644 --- a/build/docs-mcp-server/node/src/tools/get-page.ts +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -27,14 +27,9 @@ function collectCandidates(index: DocsIndex, input: GetPageInput): Page[] { if (p) byUrl.set(p.url, p); }; - if (input.url) { - const exact = index.getByUrl(input.url); - if (exact) add(exact); - else index.matchByUrlSuffix(input.url).forEach(add); - } - if (input.id) { - index.getPagesById(input.id).forEach(add); - } + // A boundary-suffix url (only reached when there was no exact match) plus id. + if (input.url) index.matchByUrlSuffix(input.url).forEach(add); + if (input.id) index.getPagesById(input.id).forEach(add); return [...byUrl.values()]; } @@ -45,29 +40,19 @@ function describeHandles(input: GetPageInput): string { return parts.join(" and "); } -/** Fetch one page, optionally filtered to sections with the given roles. */ -export function getPage(index: DocsIndex, input: GetPageInput) { - const candidates = collectCandidates(index, input); - - if (candidates.length === 0) { - return { error: `Page not found for ${describeHandles(input)}.` }; - } - if (candidates.length > 1) { - // Ambiguous (a non-unique id / boundary suffix) or conflicting (url and id - // point at different pages) — either way, make the caller pick a url. - return { - error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, - candidates: candidates.map((p) => ({ title: p.title, url: p.url })), - }; - } +function ambiguous(input: GetPageInput, candidates: Page[]) { + return { + error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, + candidates: candidates.map((p) => ({ title: p.title, url: p.url })), + }; +} - const page = candidates[0]; +function render(page: Page, input: GetPageInput) { let sections = page.sections ?? []; if (input.roles && input.roles.length) { const want = new Set(input.roles.map((r) => r.toLowerCase())); sections = sections.filter((s) => want.has((s.role ?? "").toLowerCase())); } - return { id: page.id, title: page.title, @@ -78,3 +63,37 @@ export function getPage(index: DocsIndex, input: GetPageInput) { sections, }; } + +/** Fetch one page, optionally filtered to sections with the given roles. */ +export function getPage(index: DocsIndex, input: GetPageInput) { + // An exact url is unique and authoritative. Return it directly rather than + // diluting it with a (possibly non-unique) id supplied alongside it — UNLESS + // the id points somewhere else entirely, which is a genuine conflict worth + // surfacing. A search hit's id is that page's own id, so the common + // exact-url + its-own-id case resolves cleanly. + if (input.url) { + const exact = index.getByUrl(input.url); + if (exact) { + if (input.id) { + const idPages = index.getPagesById(input.id); + if (idPages.length > 0 && !idPages.some((p) => p.url === exact.url)) { + const byUrl = new Map(); + [exact, ...idPages].forEach((p) => byUrl.set(p.url, p)); + return ambiguous(input, [...byUrl.values()]); + } + } + return render(exact, input); + } + } + + // Otherwise converge across the remaining handles (boundary-suffix url + id) + // and resolve only when they point at exactly one page. + const candidates = collectCandidates(index, input); + if (candidates.length === 0) { + return { error: `Page not found for ${describeHandles(input)}.` }; + } + if (candidates.length > 1) { + return ambiguous(input, candidates); + } + return render(candidates[0], input); +} From 4b94459261fa1f33c7e0488e4aa83f833dfd3086 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 10:54:44 +0100 Subject: [PATCH 8/9] DOC-6809 Add concept/how-to eval cases; reveals command boost is overfit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit De-bias the retrieval eval (was 22 command-only cases): tag each case command/concept and add 13 concept/how-to cases with feed-verified ground truth (incl. pages under /operate/, to test the demotion). Runner now reports recall per kind. Result under the shipped weighting: command recall@5 86% / MRR 0.65 but **concept recall@5 46% / MRR 0.29** — concept retrieval is ~half as good. The /commands/* x1.5 boost is the cause: it ranks command pages above the canonical concept page when both compete (persistence -> bgrewriteaof, replication -> cluster-replicate, keyspace-notifications -> expire), and the blanket /operate/ demotion drags down legitimate concept pages. Ablation (neutralise command boost, demote only rest-api/release-notes/references): concept @5 46->62, MRR .29->.45; command @5 86->73. Command-optimised vs balanced is a workload call — left as an open decision in SPEC, weighting unchanged for now. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 14 +++++ build/docs-mcp-server/node/README.md | 33 +++++++---- .../docs-mcp-server/node/test/eval/cases.json | 58 ++++++++++++------- build/docs-mcp-server/node/test/eval/run.mjs | 34 +++++++---- 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index 97d26837b9..c42f790dfd 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -211,6 +211,20 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting **weakened but did not eliminate** the §6 vector-search case — the eval now lets that call be made on numbers. +- **Command boost is command-overfit (found after adding concept cases).** With + 13 concept/how-to cases added, per-kind numbers diverge sharply: command + recall@5 86% / MRR 0.65 vs **concept recall@5 46% / MRR 0.29**. The + `/commands/*` ×1.5 boost is the cause — it ranks command pages above the + canonical concept page when both compete ("configure persistence" → + `bgrewriteaof`; "set up replication" → `cluster-replicate`; "keyspace + notifications" → `expire`), and the blanket `/operate/` demotion drags down + legitimate concept pages (persistence, replication). Ablation (neutralise the + command boost, demote only REST-API/release-notes/references): concept @5 + 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Open decision:** command- + optimised (current) vs balanced weighting — a workload-mix call. A modest + command boost (×1.2) helps command none vs neutral, so the command signal + should really come from better lexical handling or vectors, not the thumb on + the scale. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 2cd83d5dc6..28b6c1cd75 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -71,20 +71,29 @@ check that flags any expected url missing from the feed. The feed is read from `DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; `curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). -**Results (22 command-lookup cases):** +Cases are tagged `command` (22) or `concept` (13, how-to / concept pages) so the +runner reports recall per kind — because command and concept queries behave very +differently. -| | recall@1 | recall@3 | recall@5 | recall@10 | MRR | +**Current results (shipped config: Porter stemming + field boosts + page-type +weighting with `/commands/*` ×1.5, `/operate/` ×0.7, REST-API/release-notes ×0.5):** + +| group | recall@1 | @3 | @5 | @10 | MRR | |---|---|---|---|---|---| -| lexical baseline | 32% | 45% | 59% | 73% | 0.42 | -| + Porter stemming + page-type weighting | 50% | 68% | 86% | 95% | 0.65 | - -Stemming (append↔appends) and demoting secondary pages (release-notes / REST-API / -operator) while modestly boosting `/commands/*` lifted recall@5 from 59% → 86%. -**Caveat:** this eval is command-heavy, so the `/commands/*` boost partly flatters -it — concept/how-to query cases are not yet covered and should be added before -concluding lexical is sufficient generally. The remaining case (`del`/`unlink` -for "remove a key", beaten by `flushdb`) is a genuine lexical limitation and is -the kind of gap vector search would close (SPEC §6/§10). +| command (22) | 50% | 68% | 86% | 95% | 0.65 | +| concept (13) | 15% | 31% | 46% | 62% | 0.29 | +| overall (35) | 37% | 54% | 71% | 83% | 0.52 | + +(Un-stemmed, un-weighted lexical baseline on the command set was 59%@5 / 0.42.) + +**Finding:** command retrieval is good, but **concept/how-to retrieval is about +half as good** — and the `/commands/*` boost is the cause: it lifts command +queries but pushes command pages *above* the canonical concept page when both +compete (e.g. "configure persistence" → `bgrewriteaof`; "set up replication" → +`cluster-replicate`). An ablation neutralising the command boost moves concept +@5 46%→62% and MRR 0.29→0.45, at the cost of command @5 86%→73%. That trade-off +(command-optimised vs balanced) is a product call; the residual misses are pure +semantic gaps that motivate vector search (SPEC §6/§10). ## Known limitations (v0) diff --git a/build/docs-mcp-server/node/test/eval/cases.json b/build/docs-mcp-server/node/test/eval/cases.json index 382331819e..3e24aa50ce 100644 --- a/build/docs-mcp-server/node/test/eval/cases.json +++ b/build/docs-mcp-server/node/test/eval/cases.json @@ -1,24 +1,38 @@ [ - { "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, - { "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, - { "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, - { "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, - { "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, - { "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, - { "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, - { "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, - { "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, - { "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, - { "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, - { "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, - { "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, - { "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, - { "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, - { "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, - { "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, - { "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, - { "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, - { "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, - { "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, - { "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] } + { "kind": "command", "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, + { "kind": "command", "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, + { "kind": "command", "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, + { "kind": "command", "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, + { "kind": "command", "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, + { "kind": "command", "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, + { "kind": "command", "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, + { "kind": "command", "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, + { "kind": "command", "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, + { "kind": "command", "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, + { "kind": "command", "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, + { "kind": "command", "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, + { "kind": "command", "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, + { "kind": "command", "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, + { "kind": "command", "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, + { "kind": "command", "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, + { "kind": "command", "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, + { "kind": "command", "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, + { "kind": "command", "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, + { "kind": "command", "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, + { "kind": "command", "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, + { "kind": "command", "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] }, + + { "kind": "concept", "q": "connect to Redis from a Python application", "expected": ["https://redis.io/docs/latest/develop/clients/redis-py/connect/"] }, + { "kind": "concept", "q": "connect to Redis using the Jedis Java client", "expected": ["https://redis.io/docs/latest/develop/clients/jedis/connect/"] }, + { "kind": "concept", "q": "what are the differences between the Redis data types", "expected": ["https://redis.io/docs/latest/develop/data-types/compare-data-types/", "https://redis.io/docs/latest/develop/data-types/"] }, + { "kind": "concept", "q": "how do Redis transactions work", "expected": ["https://redis.io/docs/latest/develop/using-commands/transactions/"] }, + { "kind": "concept", "q": "send several commands together to reduce round trips", "expected": ["https://redis.io/docs/latest/develop/using-commands/pipelining/"] }, + { "kind": "concept", "q": "configure persistence with RDB snapshots and the append-only file", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/"] }, + { "kind": "concept", "q": "set up replication between a primary and its replicas", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/replication/"] }, + { "kind": "concept", "q": "run a vector similarity search over embeddings", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/query/vector-search/", "https://redis.io/docs/latest/develop/ai/search-and-query/"] }, + { "kind": "concept", "q": "get notified when keys are changed or expire", "expected": ["https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/"] }, + { "kind": "concept", "q": "cache data on the client side to reduce load", "expected": ["https://redis.io/docs/latest/develop/clients/client-side-caching/", "https://redis.io/docs/latest/develop/reference/client-side-caching/"] }, + { "kind": "concept", "q": "work with the string data type", "expected": ["https://redis.io/docs/latest/develop/data-types/strings/"] }, + { "kind": "concept", "q": "use hashes to store object-like records", "expected": ["https://redis.io/docs/latest/develop/data-types/hashes/"] }, + { "kind": "concept", "q": "full-text search and secondary indexing over Redis data", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/"] } ] diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs index c73964d8ca..6c8da79419 100644 --- a/build/docs-mcp-server/node/test/eval/run.mjs +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -43,28 +43,40 @@ for (const c of cases) { break; } } - rows.push({ q: c.q, rank, top: results[0] }); + rows.push({ kind: c.kind ?? "command", q: c.q, rank, top: results[0] }); } -const scored = rows.length; -const recall = Object.fromEntries( - K.map((k) => [k, rows.filter((r) => r.rank && r.rank <= k).length / scored]), -); -const mrr = rows.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / scored; +function metrics(set) { + const n = set.length || 1; + const recall = Object.fromEntries( + K.map((k) => [k, set.filter((r) => r.rank && r.rank <= k).length / n]), + ); + const mrr = set.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / n; + return { recall, mrr }; +} console.log( - `Feed: ${pages.length} pages | cases scored: ${scored}` + + `Feed: ${pages.length} pages | cases scored: ${rows.length}` + (broken.length ? ` | ${broken.length} BROKEN (expected url not in feed)` : "") + "\n", ); for (const r of rows) { const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; - console.log(`${tag} ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); + console.log(`${tag} [${r.kind.slice(0, 4)}] ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); } -console.log("\n--- retrieval quality ---"); -for (const k of K) console.log(`recall@${k}: ${(recall[k] * 100).toFixed(0)}%`); -console.log(`MRR: ${mrr.toFixed(3)}`); +const groups = [ + ["overall", rows], + ["command", rows.filter((r) => r.kind === "command")], + ["concept", rows.filter((r) => r.kind === "concept")], +]; +console.log("\n--- retrieval quality (recall@1 / @3 / @5 / @10 | MRR) ---"); +for (const [label, set] of groups) { + if (!set.length) continue; + const m = metrics(set); + const cells = K.map((k) => `${(m.recall[k] * 100).toFixed(0)}%`.padStart(4)).join(" / "); + console.log(`${label.padEnd(8)} (n=${String(set.length).padStart(2)}): ${cells} | ${m.mrr.toFixed(3)}`); +} if (broken.length) { console.log("\n--- BROKEN eval cases (fix ground truth) ---"); From cf2f90011207e6e510260bbf8775ce2bf0f18ed8 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Wed, 8 Jul 2026 11:00:20 +0100 Subject: [PATCH 9/9] DOC-6809 Adopt balanced page-type weighting (drop command-boost overfit) Per the concept-case eval finding, switch page weighting from command-optimised to balanced: demote only release-notes/REST-API/references (x0.5); drop the /commands/* x1.5 boost and the blanket /operate/ x0.7 demotion. This trades command recall@5 86->73 for concept 46->62 and the best overall MRR (0.53), and stops burying legitimate /operate/ concept pages (persistence, replication). Command ranking should be lifted by better signal (vectors), not the boost. Eval + smoke green; SPEC decision marked resolved. Co-Authored-By: Claude Opus 4.8 (1M context) --- build/docs-mcp-server/SPEC.md | 26 +++++++++++------------- build/docs-mcp-server/node/README.md | 26 ++++++++++++------------ build/docs-mcp-server/node/src/search.ts | 24 ++++++++++++++-------- 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md index c42f790dfd..fd78393864 100644 --- a/build/docs-mcp-server/SPEC.md +++ b/build/docs-mcp-server/SPEC.md @@ -202,15 +202,13 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. well-structured the corpus already is? - **Ranking quality (measured via the eval harness):** lexical BM25 with Porter stemming, stopword removal, title/summary/slug field boosts, and - page-type weighting (demote release-notes/REST-API/operator, modestly boost - `/commands/*`) gets **recall@5 86% / MRR 0.65** on 22 command-lookup cases — - up from a **59% / 0.42** un-stemmed, un-weighted baseline. Two caveats: (1) - the eval is command-heavy so the `/commands/*` boost partly flatters it — - **add concept/how-to eval cases** before concluding lexical is sufficient - generally; (2) the residual misses are pure semantic gaps ("remove a key" → - `flushdb` beats `del`) that only vector search closes. Net: stemming+weighting - **weakened but did not eliminate** the §6 vector-search case — the eval now - lets that call be made on numbers. + balanced page-type weighting (demote release-notes/REST-API/references only) + gets, on the 35-case eval (22 command + 13 concept), **command recall@5 73% / + concept 62% / overall 69%, MRR 0.53** — up from a **59% / 0.42** un-stemmed, + un-weighted lexical baseline on the command set. Residual misses are pure + semantic gaps ("remove a key" → `flushdb` beats `del`) that only vector search + closes. Net: stemming+weighting **weakened but did not eliminate** the §6 + vector-search case — the eval now lets that call be made on numbers. - **Command boost is command-overfit (found after adding concept cases).** With 13 concept/how-to cases added, per-kind numbers diverge sharply: command recall@5 86% / MRR 0.65 vs **concept recall@5 46% / MRR 0.29**. The @@ -220,11 +218,11 @@ vector-on-Redis as a later upgrade to the **hosted** endpoint only. notifications" → `expire`), and the blanket `/operate/` demotion drags down legitimate concept pages (persistence, replication). Ablation (neutralise the command boost, demote only REST-API/release-notes/references): concept @5 - 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Open decision:** command- - optimised (current) vs balanced weighting — a workload-mix call. A modest - command boost (×1.2) helps command none vs neutral, so the command signal - should really come from better lexical handling or vectors, not the thumb on - the scale. + 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Resolved: adopted the balanced + weighting** (no command boost, no blanket `/operate/` demotion). A modest + command boost (×1.2) helped command none vs neutral, so lifting command + ranking should come from better lexical handling or vectors, not a bigger + thumb on the scale — the next lever, measured against this eval. - **Section-role vocabulary (found via live MCP test):** the roles the spec assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match the feed. Command pages actually carry `content` / `parameters` / `example` diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md index 28b6c1cd75..b95e0f761a 100644 --- a/build/docs-mcp-server/node/README.md +++ b/build/docs-mcp-server/node/README.md @@ -75,25 +75,25 @@ Cases are tagged `command` (22) or `concept` (13, how-to / concept pages) so the runner reports recall per kind — because command and concept queries behave very differently. -**Current results (shipped config: Porter stemming + field boosts + page-type -weighting with `/commands/*` ×1.5, `/operate/` ×0.7, REST-API/release-notes ×0.5):** +**Current results (shipped config: Porter stemming + field boosts + *balanced* +page-type weighting — demote REST-API/release-notes/references ×0.5, no command +boost, no blanket `/operate/` demotion):** | group | recall@1 | @3 | @5 | @10 | MRR | |---|---|---|---|---|---| -| command (22) | 50% | 68% | 86% | 95% | 0.65 | -| concept (13) | 15% | 31% | 46% | 62% | 0.29 | -| overall (35) | 37% | 54% | 71% | 83% | 0.52 | +| command (22) | 41% | 64% | 73% | 95% | 0.57 | +| concept (13) | 31% | 54% | 62% | 77% | 0.45 | +| overall (35) | 37% | 60% | 69% | 89% | 0.53 | (Un-stemmed, un-weighted lexical baseline on the command set was 59%@5 / 0.42.) -**Finding:** command retrieval is good, but **concept/how-to retrieval is about -half as good** — and the `/commands/*` boost is the cause: it lifts command -queries but pushes command pages *above* the canonical concept page when both -compete (e.g. "configure persistence" → `bgrewriteaof`; "set up replication" → -`cluster-replicate`). An ablation neutralising the command boost moves concept -@5 46%→62% and MRR 0.29→0.45, at the cost of command @5 86%→73%. That trade-off -(command-optimised vs balanced) is a product call; the residual misses are pure -semantic gaps that motivate vector search (SPEC §6/§10). +**Why balanced:** an earlier command-optimised config (`/commands/*` ×1.5, +`/operate/` ×0.7) scored command @5 86% but only concept @5 46% — the boost +ranked command pages above the canonical concept page when both competed +("configure persistence" → `bgrewriteaof`). We chose the balanced weighting: +concept @5 46%→62% for command @5 86%→73% (SPEC §10). The residual misses are +pure semantic gaps that motivate vector search (SPEC §6/§10) — the right lever +for lifting both, rather than a bigger thumb on the scale. ## Known limitations (v0) diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts index d5d913285d..c0ad9ae710 100644 --- a/build/docs-mcp-server/node/src/search.ts +++ b/build/docs-mcp-server/node/src/search.ts @@ -8,17 +8,23 @@ import { stem } from "./stem.js"; const K1 = 1.5; const B = 0.75; -// Page-type weighting applied to the final score. Command reference pages are -// the canonical answer for command-shaped queries; release-notes / REST-API / -// operator pages are secondary and were observed outranking primary docs -// (e.g. "remove a key" -> operate/.../remove-node). Multipliers, not filters. -// NOTE: the retrieval eval is command-heavy, so keep the command boost modest -// to avoid tuning to the eval — the demotions are the more general fix. +// Page-type weighting applied to the final score. Demote clearly-secondary +// reference material (release-notes / REST-API / other references) that was +// observed outranking primary docs. Multipliers, not filters. +// +// Deliberately balanced, NOT command-optimised: an earlier config boosted +// /commands/* (x1.5) and demoted all of /operate/ (x0.7), which lifted command +// queries but ranked command pages above the canonical concept page when both +// competed (persistence -> bgrewriteaof) and buried legitimate /operate/ +// concept pages. The eval showed that cost concept recall@5 ~16pts for ~13pts +// of command gain, so we chose the balanced weighting (SPEC §10). Lifting +// command ranking further should come from better signal (vectors), not a +// bigger thumb on the scale. function pageWeight(url: string): number { const u = url.toLowerCase(); - if (u.includes("/release-notes") || u.includes("/rest-api/")) return 0.5; - if (u.includes("/operate/")) return 0.7; - if (u.includes("/commands/")) return 1.5; + if (u.includes("/release-notes") || u.includes("/rest-api/") || u.includes("/references/")) { + return 0.5; + } return 1; }