diff --git a/build/docs-mcp-server/SPEC.md b/build/docs-mcp-server/SPEC.md new file mode 100644 index 0000000000..fd78393864 --- /dev/null +++ b/build/docs-mcp-server/SPEC.md @@ -0,0 +1,262 @@ +# Docs MCP server — design spec + +**Status:** Draft (investigation, DOC-6809) +**Owner:** Docs +**Scope:** A read-only MCP server that lets AI coding agents query the Redis +documentation corpus as a tool, backed by the JSON/NDJSON feed we already +publish. + +--- + +## 1. Motivation + +We already publish AI-readable docs three ways: `llms.txt` (index), per-page +Markdown (`index.html.md`), and structured JSON/NDJSON with role-tagged +sections. These are all *passive files* — an agent must know they exist, fetch +them, and do its own retrieval. + +The gap is an *active, queryable* surface: a tool an agent (Cursor, Claude +Code, ChatGPT, VS Code) can call mid-task to get a **current, sourced** answer +instead of relying on stale training data. That is what this server provides. + +### Explicitly *not* this server + +- **`redis/mcp-redis`** is a *data-plane* server: it connects an agent to a + *running Redis instance* to read/write/query data. It needs a connection + string and can mutate data. +- **This server** is a *knowledge-plane* server: it connects an agent to the + *documentation*. It is read-only, needs no database, no credentials, and its + entire value is returning citations to our docs. + +They are orthogonal and should stay separate products/installs. Bundling doc +lookup into the data-plane server forces the reference-only audience to stand +up a data server and hand it credentials — friction that kills adoption for +the exact audience (coding agents) that benefits most. + +## 2. Goals / non-goals + +**Goals** +- Thin retrieval wrapper over the **existing** JSON feed — no new content + pipeline. +- Read-only, no secrets, no live DB connection. +- Every response carries a canonical `url` (citations by construction). +- Token-lean: search returns summaries + refs; agents drill down deliberately. +- Version-aware (our docs are versioned; mixing versions is a correctness bug). + +**Non-goals** +- No writes, no code execution, no live Redis access (that's `mcp-redis`). +- No new authoring format — we consume `sections[]` / `examples[]` as-is. +- WebMCP / in-browser tool registration — out of scope for now (different + layer, different audience; see DOC-6809 discussion). + +## 3. Data source + +No new pipeline. Reuse the current build output: + +``` +Hugo ──► per-page public/**/index.json ──► generate_ndjson.py ──► docs.ndjson +``` + +Document schema (already published on the *AI Agent Resources* page): + +- **Page**: `id`, `title`, `url`, `summary`, `page_type` (`content` | `index`), + `content_hash`, `sections[]`, `examples[]`, `children[]` +- **Section**: `id`, `title`, `role` (`overview` | `syntax` | `parameters` | + `returns` | `example` | …), `text` +- **Example**: `id`, `language`, `code`, `section_id` + +The server loads `docs.ndjson` (or an index built from it) at startup. Because +`content_hash` is deterministic (`sha256` over summary + section text + +example code), it doubles as a cache/freshness key. + +## 4. Tool surface + +Five tools. Names, inputs, and the field of the existing schema each is +projected from: + +### `search_docs` +Rank pages by relevance. Returns **refs only, no full text**. + +- **In:** `query` (string, required); optional `page_type`, `group`, + `version`, `limit` (default 10) +- **Out:** `[{ id, title, url, summary, matching_section_ids[] }]` +- **From:** NDJSON feed; `page_type` filter lets callers skip `index` pages. + +### `get_page` +Fetch one page, optionally filtered to specific section roles. + +- **In:** `id` **or** `url` (required); optional `roles[]` (e.g. + `["syntax","parameters"]`) +- **Out:** `{ id, title, url, summary, page_type, content_hash, sections[] }` + where `sections` is filtered to `roles[]` if given +- **From:** per-page `index.json`. `roles[]` filtering is only possible because + sections are role-tagged — big token savings (pull `parameters` without the + overview prose). + +### `get_section` +Return a single role-tagged chunk — the retrieval-native unit. + +- **In:** `page_id` (required), `section_id` (required) +- **Out:** `{ page_id, section_id, title, role, text, url }` +- **From:** `sections[]`. + +### `get_examples` +Return runnable code, filterable by language. **The highest-value tool for +coding agents** — their most common need is "the go-redis snippet for `XADD`", +not prose. + +- **In:** `query` **or** `command` (one required); optional `language` + (`python` | `go` | `java` | …) +- **Out:** `[{ id, code, language, url, section_id }]` +- **From:** `examples[]` (carries `language` + `section_id` already). + +### `get_command` +Convenience lookup for the highest-traffic page type. + +- **In:** `name` (e.g. `XADD`) +- **Out:** command page with `syntax`, `parameters`, `returns` sections + + `examples[]` + `url` +- **From:** command pages (specialised `get_page`; commands are first-class). + +## 5. Response conventions + +- **Always include `url`.** Agents cite; users click. +- **Search never returns full text.** Force the drill-down path + (`search_docs` → `get_section` / `get_examples`) so context stays small. +- **Return `content_hash` on page/section responses.** Lets agents and our own + eval harness do `If-None-Match`-style freshness checks for free. +- **Truncate defensively.** Cap `text` length per section in responses; expose + a `truncated: true` flag rather than silently cutting. + +## 6. Transport & deployment + +Follow the existing repo pattern (`build/command_api_mapping/mcp-server/`): +TypeScript, `@modelcontextprotocol/sdk`, Zod input schemas. + +- **Local / stdio:** publish an npx-runnable package so developers can add it to + Cursor/Claude Code config. Zero infra. +- **Remote / hosted:** an HTTP+SSE endpoint on `redis.io` (e.g. + `https://redis.io/mcp`) built from the same handlers, so no install is + required. Advertise it on the *AI Agent Resources* page next to `llms.txt`. + +Both modes share one core: load feed → build index → handle tool calls. The +data is public, so the remote endpoint needs no auth (rate-limit only). + +### Search backend vs. transport — what needs a datastore + +Whether the server needs a Redis (or any) backend depends entirely on the +search implementation, **not** on the transport. The corpus is small +(`docs.ndjson` ≈ 30 MB / ≈ 5 MB gzipped, ≈ 4,100 docs), which is what makes +the lexical path infra-free. + +| | Lexical (BM25) | Vector (semantic) | +|---|---|---| +| **stdio (client-side)** | ✅ self-contained in-memory index | ❌ impractical (would ship an index + embedding model per install) | +| **remote (hosted)** | ✅ in-process index, no datastore | ✅ needs a vector store (RediSearch / RedisVL) | + +- **Lexical (v1):** build a BM25 index in process memory from `docs.ndjson` at + startup — MiniSearch/Lunr (JS) or a simple BM25 (Python). No datastore, even + when hosted; horizontally-scaled instances each just load ~5 MB gzipped at + boot and build their own index. Role-tagged sections and exact command-name + tokens make lexical retrieval unusually strong on this corpus. +- **Vector (v2, hosted only):** needs embeddings computed offline at build + time, a vector index queried at runtime, and query-time embedding on each + request. That implies a real server-side backend — the natural fit is + **RediSearch / RedisVL**, which doubles as a Redis showcase. It cannot run + purely client-side, so it's a hosted-endpoint upgrade, not a stdio feature. + +Recommendation: ship v1 lexical in **both** modes with no backend; treat +vector-on-Redis as a later upgrade to the **hosted** endpoint only. + +## 7. Versioning + +- **Planned:** `search_docs` / `get_page` accept an optional `version` (default + `latest`), and responses echo the resolved version so an agent can't silently + blend versions — the single most common RAG-over-docs correctness bug. +- **v0 status: deferred, not implemented.** The prototype does **not** expose a + `version` param. An earlier v0 advertised a `latest` default it didn't + enforce (the filter was a no-op, and the live feed is single-version anyway), + which misleads agents. Rather than bake in a `page.url.includes("//")` + heuristic, the param was removed until a committed URL/version model exists + (Bugbot #3585 + Codex review). Add it back with the real filter when the feed + carries multiple version trees. + +## 8. Freshness + +- Rebuild the server's index whenever `docs.ndjson` is regenerated (same build + step). No separate content pipeline to keep in sync. +- `content_hash` per page enables incremental index updates and client caching. + +## 9. Security + +- Read-only. No write tools, no code execution, no connection string. +- Serves only already-public content. +- Remote endpoint: rate-limit, no auth, no PII. + +## 10. Open questions + +- **Search backend:** resolved for v1 — lexical (BM25 over NDJSON), no + datastore, runs in both stdio and hosted modes (see §6). Open: do we add + vector search as a v2 upgrade to the hosted endpoint, backed by RediSearch / + RedisVL, and is the retrieval gain worth the added infra given how + well-structured the corpus already is? +- **Ranking quality (measured via the eval harness):** lexical BM25 with + Porter stemming, stopword removal, title/summary/slug field boosts, and + balanced page-type weighting (demote release-notes/REST-API/references only) + gets, on the 35-case eval (22 command + 13 concept), **command recall@5 73% / + concept 62% / overall 69%, MRR 0.53** — up from a **59% / 0.42** un-stemmed, + un-weighted lexical baseline on the command set. Residual misses are pure + semantic gaps ("remove a key" → `flushdb` beats `del`) that only vector search + closes. Net: stemming+weighting **weakened but did not eliminate** the §6 + vector-search case — the eval now lets that call be made on numbers. +- **Command boost is command-overfit (found after adding concept cases).** With + 13 concept/how-to cases added, per-kind numbers diverge sharply: command + recall@5 86% / MRR 0.65 vs **concept recall@5 46% / MRR 0.29**. The + `/commands/*` ×1.5 boost is the cause — it ranks command pages above the + canonical concept page when both compete ("configure persistence" → + `bgrewriteaof`; "set up replication" → `cluster-replicate`; "keyspace + notifications" → `expire`), and the blanket `/operate/` demotion drags down + legitimate concept pages (persistence, replication). Ablation (neutralise the + command boost, demote only REST-API/release-notes/references): concept @5 + 46%→62%, MRR 0.29→0.45; command @5 86%→73%. **Resolved: adopted the balanced + weighting** (no command boost, no blanket `/operate/` demotion). A modest + command boost (×1.2) helped command none vs neutral, so lifting command + ranking should come from better lexical handling or vectors, not a bigger + thumb on the scale — the next lever, measured against this eval. +- **Section-role vocabulary (found via live MCP test):** the roles the spec + assumed (`syntax`, `parameters`, `returns`, `example`) do **not** all match + the feed. Command pages actually carry `content` / `parameters` / `example` + (singular) / `returns` — there is no `syntax` role, and it's `example` not + `examples`. So a `roles: ["examples","syntax"]` filter returns **zero + sections** against the real feed (verified on EXPIREAT). Before building + `get_examples` / `get_command` and documenting `roles`, enumerate the actual + role set across the corpus and align tool params/docs to it (and decide + whether the server should normalise synonyms like `examples`→`example`). +- **`get_command` coverage:** command pages *do* carry `parameters`/`returns`/ + `example` roles (confirmed: `get_page('expire')` → roles + `[content, parameters, example, returns]`). Confirm this holds across all + command pages or add a fallback. +- **Package ownership:** does this live here in `docs`, or graduate to its own + repo like `mcp-redis`? +- **Overlap with `mcp-redis`:** does that server already do any doc lookup we + should pull into here and deprecate there? + +## 11. Phased plan + +1. **v0 (prototype):** stdio server, lexical `search_docs` + `get_page` over a + local `docs.ndjson`. Prove the loop in Claude Code. +2. **v1:** add `get_examples`, `get_section`, `get_command`; `version` support; + token-budget guards. +3. **v2:** hosted remote endpoint on `redis.io`; advertise on AI Agent + Resources. +4. **Ongoing:** wire an **AI-answer eval** — a fixed set of real questions run + through the server, scored for correctness — as a docs-quality regression + gate. (Extends the code-example verification mindset to answer quality.) + +## 12. Success signals + +- A coding agent, given only this server, answers common Redis how-to questions + correctly and with citations. +- Measurable reduction in version-mixing / stale-API answers versus the model's + own training data. +- Adoption: entries in Cursor/Claude Code MCP configs pointing at the endpoint. diff --git a/build/docs-mcp-server/node/.gitignore b/build/docs-mcp-server/node/.gitignore new file mode 100644 index 0000000000..d458242e86 --- /dev/null +++ b/build/docs-mcp-server/node/.gitignore @@ -0,0 +1,4 @@ +node_modules/ +dist/ +*.log +test/eval/docs.ndjson* diff --git a/build/docs-mcp-server/node/README.md b/build/docs-mcp-server/node/README.md new file mode 100644 index 0000000000..b95e0f761a --- /dev/null +++ b/build/docs-mcp-server/node/README.md @@ -0,0 +1,108 @@ +# redis-docs-mcp (v0 prototype) + +A read-only MCP server that lets an AI coding agent query the Redis +documentation corpus as a tool, backed by the `docs.ndjson` feed we already +publish. See [`../SPEC.md`](../SPEC.md) for the full design. + +**This is a v0 prototype.** It ships two tools (`search_docs`, `get_page`) over +an in-memory lexical (BM25 + field-boost) index. No datastore, no credentials, +no live Redis connection. + +## Install & build + +```bash +cd build/docs-mcp-server/node +npm install +npm run build # compiles to dist/ +``` + +## Try it offline (no network) + +```bash +npm run smoke # runs against test/fixture.ndjson +``` + +Point it at the real feed (or any local `.ndjson` / `.ndjson.gz`): + +```bash +DOCS_NDJSON="https://redis.io/docs/latest/docs.ndjson" npm run smoke +``` + +## Run as an MCP server + +The server speaks MCP over stdio. Feed source is set via `DOCS_NDJSON` +(default: `https://redis.io/docs/latest/docs.ndjson`). + +Add to a Claude Code / Cursor MCP config after `npm run build`: + +```json +{ + "mcpServers": { + "redis-docs": { + "command": "node", + "args": ["/ABSOLUTE/PATH/build/docs-mcp-server/node/dist/index.js"], + "env": { "DOCS_NDJSON": "https://redis.io/docs/latest/docs.ndjson" } + } + } +} +``` + +## Tools + +| Tool | Inputs | Returns | +|------|--------|---------| +| `search_docs` | `query` (req), `page_type`, `limit` | ranked `[{id, title, url, summary, page_type, score, matching_section_ids}]` — refs only, no full text | +| `get_page` | `id` **or** `url` (req), `roles[]` | one page with `content_hash` + `sections`, optionally filtered to the given section roles | + +Typical agent flow: `search_docs` → pick a result → `get_page` with `roles` +(e.g. `["parameters","returns"]`) to pull just what's needed. + +## Measured (real feed, ~2,530 pages) + +- Index build: ~0.3 s. Query latency: ~75–125 ms. This is why v0 needs no + datastore and why Rust/WASM would be premature (see SPEC §6). + +## Retrieval eval + +`npm run eval` scores retrieval quality: it runs the questions in +`test/eval/cases.json` (command-lookup questions phrased *without* the command +name) through `search_docs` and reports recall@k / MRR, plus a data-integrity +check that flags any expected url missing from the feed. The feed is read from +`DOCS_NDJSON` or a local cache at `test/eval/docs.ndjson.gz` (gitignored; +`curl -o test/eval/docs.ndjson.gz https://redis.io/docs/latest/docs.ndjson.gz`). + +Cases are tagged `command` (22) or `concept` (13, how-to / concept pages) so the +runner reports recall per kind — because command and concept queries behave very +differently. + +**Current results (shipped config: Porter stemming + field boosts + *balanced* +page-type weighting — demote REST-API/release-notes/references ×0.5, no command +boost, no blanket `/operate/` demotion):** + +| group | recall@1 | @3 | @5 | @10 | MRR | +|---|---|---|---|---|---| +| command (22) | 41% | 64% | 73% | 95% | 0.57 | +| concept (13) | 31% | 54% | 62% | 77% | 0.45 | +| overall (35) | 37% | 60% | 69% | 89% | 0.53 | + +(Un-stemmed, un-weighted lexical baseline on the command set was 59%@5 / 0.42.) + +**Why balanced:** an earlier command-optimised config (`/commands/*` ×1.5, +`/operate/` ×0.7) scored command @5 86% but only concept @5 46% — the boost +ranked command pages above the canonical concept page when both competed +("configure persistence" → `bgrewriteaof`). We chose the balanced weighting: +concept @5 46%→62% for command @5 86%→73% (SPEC §10). The residual misses are +pure semantic gaps that motivate vector search (SPEC §6/§10) — the right lever +for lifting both, rather than a bigger thumb on the scale. + +## Known limitations (v0) + +- **Ranking is lexical (BM25 + Porter stemming + field/page-type weighting).** + Now recall@5 86% on command lookups (see eval above), but still lexical: it + can't bridge pure semantic gaps (e.g. "remove a key" → `flushdb` over `del`), + and concept/how-to queries are unmeasured. Closing the remainder is the case + for vector search (SPEC §6/§10). +- **No version filtering.** The `version` param was removed until a committed + URL/version model exists (the feed is single-version today); see SPEC §7. +- Only `search_docs` + `get_page`. `get_examples`, `get_section`, + `get_command` are v1 (SPEC §4/§11). diff --git a/build/docs-mcp-server/node/package.json b/build/docs-mcp-server/node/package.json new file mode 100644 index 0000000000..90f51e6444 --- /dev/null +++ b/build/docs-mcp-server/node/package.json @@ -0,0 +1,32 @@ +{ + "name": "redis-docs-mcp", + "version": "0.0.1", + "description": "Read-only MCP server that queries the Redis documentation corpus (docs.ndjson) as a tool. v0 prototype: lexical search_docs + get_page.", + "type": "module", + "main": "dist/index.js", + "bin": { + "redis-docs-mcp": "dist/index.js" + }, + "scripts": { + "build": "tsc", + "start": "tsx src/index.ts", + "dev": "tsx watch src/index.ts", + "smoke": "tsx src/smoke.ts", + "eval": "npm run build && node test/eval/run.mjs" + }, + "keywords": [ + "redis", + "mcp", + "docs" + ], + "license": "MIT", + "dependencies": { + "@modelcontextprotocol/sdk": "^1.0.0", + "zod": "^3.22.0" + }, + "devDependencies": { + "@types/node": "^20.0.0", + "tsx": "^4.0.0", + "typescript": "^5.0.0" + } +} diff --git a/build/docs-mcp-server/node/src/feed.ts b/build/docs-mcp-server/node/src/feed.ts new file mode 100644 index 0000000000..7eb01a0a78 --- /dev/null +++ b/build/docs-mcp-server/node/src/feed.ts @@ -0,0 +1,44 @@ +import { readFile } from "node:fs/promises"; +import { gunzipSync } from "node:zlib"; +import type { Page } from "./types.js"; + +/** Load raw feed bytes from a local path or http(s) URL, gunzipping if .gz. */ +async function loadFeedRaw(source: string): Promise { + let buf: Buffer; + if (/^https?:\/\//i.test(source)) { + const res = await fetch(source); + if (!res.ok) { + throw new Error(`Failed to fetch feed ${source}: ${res.status} ${res.statusText}`); + } + buf = Buffer.from(await res.arrayBuffer()); + } else { + buf = await readFile(source); + } + if (source.toLowerCase().endsWith(".gz")) { + buf = gunzipSync(buf); + } + return buf.toString("utf-8"); +} + +/** Parse NDJSON text into pages, skipping blank/invalid lines and non-doc objects. */ +export function parseNdjson(text: string): Page[] { + const pages: Page[] = []; + for (const line of text.split("\n")) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const obj = JSON.parse(trimmed); + // Match the same guard generate_ndjson.py uses: must have id + title + url. + if (obj && typeof obj.id === "string" && typeof obj.url === "string" && typeof obj.title === "string") { + pages.push(obj as Page); + } + } catch { + // Not our format — skip. + } + } + return pages; +} + +export async function loadFeed(source: string): Promise { + return parseNdjson(await loadFeedRaw(source)); +} diff --git a/build/docs-mcp-server/node/src/index.ts b/build/docs-mcp-server/node/src/index.ts new file mode 100644 index 0000000000..21fbcd51ec --- /dev/null +++ b/build/docs-mcp-server/node/src/index.ts @@ -0,0 +1,103 @@ +#!/usr/bin/env node +import { Server } from "@modelcontextprotocol/sdk/server/index.js"; +import { + ListToolsRequestSchema, + CallToolRequestSchema, +} from "@modelcontextprotocol/sdk/types.js"; +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; + +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs, SearchDocsInput } from "./tools/search-docs.js"; +import { getPage, GetPageInput } from "./tools/get-page.js"; +import { toolResult, fail } from "./response.js"; + +// Feed source: local path or http(s) URL, gzip-aware. Defaults to production. +const FEED_SOURCE = + process.env.DOCS_NDJSON ?? "https://redis.io/docs/latest/docs.ndjson"; + +const TOOLS = [ + { + name: "search_docs", + description: + "Search the Redis documentation and return the most relevant pages as references (title, url, summary, matching section ids). Each hit includes a unique `url` — pass that `url` to get_page (a hit's `id` is NOT unique and may be ambiguous). Returns no full text — follow up with get_page to read a result.", + inputSchema: { + type: "object" as const, + properties: { + query: { type: "string", description: "Search query" }, + page_type: { + type: "string", + enum: ["content", "index"], + description: "Restrict to prose ('content') or navigation ('index') pages", + }, + limit: { + type: "number", + description: "Max results (default 10, max 50)", + }, + }, + required: ["query"], + }, + }, + { + name: "get_page", + description: + "Fetch a single documentation page. Prefer the unique `url` from a search_docs hit. `id` also works but is NOT unique, so an ambiguous id returns an error listing candidate urls (likewise an ambiguous partial url). Optionally filter to sections with specific roles (e.g. ['syntax','parameters']) to save tokens. Includes content_hash for caching.", + inputSchema: { + type: "object" as const, + properties: { + id: { + type: "string", + description: "Page id (last URL slug); not unique — prefer url", + }, + url: { + type: "string", + description: "Full page URL (preferred — unique). A partial/suffix URL also works when it matches exactly one page.", + }, + roles: { + type: "array", + items: { type: "string" }, + description: "Only return sections with these roles", + }, + }, + }, + }, +]; + +async function main() { + const pages = await loadFeed(FEED_SOURCE); + const index = new DocsIndex(pages); + // Log to stderr — stdout is reserved for the MCP protocol. + console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); + + const server = new Server( + { name: "redis-docs-mcp", version: "0.0.1" }, + { capabilities: { tools: {} } }, + ); + + server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: TOOLS })); + + server.setRequestHandler(CallToolRequestSchema, async (req) => { + const { name, arguments: args } = req.params; + try { + switch (name) { + case "search_docs": + return toolResult(searchDocs(index, SearchDocsInput.parse(args ?? {}))); + case "get_page": + return toolResult(getPage(index, GetPageInput.parse(args ?? {}))); + default: + return fail(`Unknown tool: ${name}`); + } + } catch (e) { + return fail(e instanceof Error ? e.message : String(e)); + } + }); + + const transport = new StdioServerTransport(); + await server.connect(transport); + console.error("[redis-docs-mcp] ready on stdio"); +} + +main().catch((e) => { + console.error("[redis-docs-mcp] fatal:", e); + process.exit(1); +}); diff --git a/build/docs-mcp-server/node/src/response.ts b/build/docs-mcp-server/node/src/response.ts new file mode 100644 index 0000000000..6f1db590f1 --- /dev/null +++ b/build/docs-mcp-server/node/src/response.ts @@ -0,0 +1,24 @@ +// MCP tool-response helpers, factored out of index.ts so tests can import them +// without triggering index.ts's main() (which starts the stdio server). + +/** + * Serialise a tool result. If the tool returned an object carrying an `error` + * field (e.g. get_page couldn't resolve the page, or an id/url was ambiguous), + * mark the MCP response as an error so clients don't treat a failed lookup as + * success. + */ +export function toolResult(data: unknown) { + const isError = typeof data === "object" && data !== null && "error" in data; + return { + content: [{ type: "text" as const, text: JSON.stringify(data, null, 2) }], + ...(isError ? { isError: true } : {}), + }; +} + +/** For protocol-level failures (unknown tool, input parse errors). */ +export function fail(message: string) { + return { + content: [{ type: "text" as const, text: JSON.stringify({ error: message }) }], + isError: true, + }; +} diff --git a/build/docs-mcp-server/node/src/search.ts b/build/docs-mcp-server/node/src/search.ts new file mode 100644 index 0000000000..c0ad9ae710 --- /dev/null +++ b/build/docs-mcp-server/node/src/search.ts @@ -0,0 +1,227 @@ +import type { Page } from "./types.js"; +import { stem } from "./stem.js"; + +// Self-contained BM25 lexical index. No external search dependency: at +// ~4,100 docs the whole index builds in-memory in well under a second, which +// is why v0 needs no datastore (see SPEC.md §6). + +const K1 = 1.5; +const B = 0.75; + +// Page-type weighting applied to the final score. Demote clearly-secondary +// reference material (release-notes / REST-API / other references) that was +// observed outranking primary docs. Multipliers, not filters. +// +// Deliberately balanced, NOT command-optimised: an earlier config boosted +// /commands/* (x1.5) and demoted all of /operate/ (x0.7), which lifted command +// queries but ranked command pages above the canonical concept page when both +// competed (persistence -> bgrewriteaof) and buried legitimate /operate/ +// concept pages. The eval showed that cost concept recall@5 ~16pts for ~13pts +// of command gain, so we chose the balanced weighting (SPEC §10). Lifting +// command ranking further should come from better signal (vectors), not a +// bigger thumb on the scale. +function pageWeight(url: string): number { + const u = url.toLowerCase(); + if (u.includes("/release-notes") || u.includes("/rest-api/") || u.includes("/references/")) { + return 0.5; + } + return 1; +} + +// Field boosts (added on top of the body BM25 score, weighted by term idf). +// A query term appearing in the title/slug/summary is a strong signal that the +// page is *about* that term — this lifts canonical command pages (whose summary +// is a one-line definition) above long pages that merely mention the terms. +const W_SUMMARY = 6; // canonical one-line definition — strongest signal +const W_TITLE = 4; +const W_SLUG = 2; // lowest: slug word-collisions (set-up-redis, key-specs) mislead + +// Common words carry no topical signal and their title/slug/summary collisions +// distort ranking (e.g. "set"/"key"). Dropped from the query only. +const STOPWORDS = new Set([ + "a", "an", "the", "to", "of", "in", "on", "for", "with", "and", "or", "is", + "are", "how", "do", "i", "my", "me", "can", "what", "when", "which", "you", + "your", "it", "this", "that", "from", "by", "as", "at", "be", "using", "use", +]); + +/** Split on non-alphanumerics so "JSON.SET" -> ["json","set"], "XADD" -> ["xadd"]. */ +function tokenize(text: string): string[] { + return text.toLowerCase().match(/[a-z0-9]+/g) ?? []; +} + +/** Tokenize + stem. Used for everything indexed and for query terms, so word + * forms conflate. Stopwords are filtered on RAW tokens before this (see search). */ +function analyze(text: string): string[] { + return tokenize(text).map(stem); +} + +function normalizeUrl(u: string): string { + return u.trim().toLowerCase().replace(/\/+$/, ""); +} + +/** Everything worth matching against for a page: slug, title, summary, section text. */ +function searchableText(p: Page): string { + const parts: string[] = [p.id ?? "", p.title ?? "", p.summary ?? ""]; + for (const s of p.sections ?? []) { + parts.push(s.title ?? "", s.text ?? ""); + } + return parts.join(" "); +} + +/** Section ids whose title/text contain any query term (capped for token budget). */ +function matchingSections(p: Page, qterms: Set): string[] { + const out: string[] = []; + for (const s of p.sections ?? []) { + const toks = new Set(analyze(`${s.title ?? ""} ${s.text ?? ""}`)); + for (const t of qterms) { + if (toks.has(t)) { + out.push(s.id); + break; + } + } + if (out.length >= 5) break; + } + return out; +} + +export interface SearchOptions { + limit?: number; + pageType?: string; +} + +export interface SearchHit { + id: string; + title: string; + url: string; + summary: string; + page_type: string; + score: number; + matching_section_ids: string[]; +} + +export class DocsIndex { + readonly pages: Page[]; + // The feed's `id` is the last URL path segment (e.g. "config", "acl") and is + // NOT unique — ~200 ids map to several pages. So id -> list, and callers must + // disambiguate by url. `url` IS unique, so byUrl stays 1:1. + private byId = new Map(); + private byUrl = new Map(); + private docs: Array<{ + page: Page; + tf: Map; + len: number; + titleTok: Set; + slugTok: Set; + summaryTok: Set; + }> = []; + private df = new Map(); + private avgdl = 0; + private N = 0; + + constructor(pages: Page[]) { + this.pages = pages; + let totalLen = 0; + for (const p of pages) { + const bucket = this.byId.get(p.id); + if (bucket) bucket.push(p); + else this.byId.set(p.id, [p]); + if (p.url) this.byUrl.set(normalizeUrl(p.url), p); + + const tokens = analyze(searchableText(p)); + if (tokens.length === 0) continue; + const tf = new Map(); + for (const t of tokens) tf.set(t, (tf.get(t) ?? 0) + 1); + for (const t of tf.keys()) this.df.set(t, (this.df.get(t) ?? 0) + 1); + this.docs.push({ + page: p, + tf, + len: tokens.length, + titleTok: new Set(analyze(p.title ?? "")), + slugTok: new Set(analyze(p.id ?? "")), + summaryTok: new Set(analyze(p.summary ?? "")), + }); + totalLen += tokens.length; + } + this.N = this.docs.length; + this.avgdl = this.N ? totalLen / this.N : 0; + } + + get size(): number { + return this.pages.length; + } + + /** All pages sharing this id (usually one, but the feed's id is not unique). */ + getPagesById(id: string): Page[] { + return this.byId.get(id) ?? []; + } + + getByUrl(url: string): Page | undefined { + return this.byUrl.get(normalizeUrl(url)); + } + + /** + * Fallback lookup when a caller passes a path or partial URL. Returns EVERY + * page whose url ends with the given suffix **at a path-segment boundary**, + * so "get" matches ".../commands/get" but NOT ".../config-get" or + * ".../arget". A suffix can still match several pages (e.g. "/install/" or a + * bare last segment shared by many pages), so the caller must disambiguate. + */ + matchByUrlSuffix(url: string): Page[] { + const target = normalizeUrl(url).replace(/^https?:\/\/[^/]+/, ""); + if (!target) return []; + // Anchor the leading edge to a "/" so we match whole path segments. The + // trailing edge is already anchored: normalizeUrl strips the trailing slash + // and we compare against the end of the string. + const anchored = target.startsWith("/") ? target : `/${target}`; + return this.pages.filter((p) => normalizeUrl(p.url).endsWith(anchored)); + } + + search(query: string, opts: SearchOptions = {}): SearchHit[] { + // Filter stopwords on RAW tokens (before stemming), then stem + dedupe. + const raw = [...new Set(tokenize(query))]; + let kept = raw.filter((t) => !STOPWORDS.has(t)); + if (kept.length === 0) kept = raw; // query was all stopwords + const qterms = [...new Set(kept.map(stem))]; + if (qterms.length === 0) return []; + + const idf = new Map(); + for (const t of qterms) { + const df = this.df.get(t) ?? 0; + idf.set(t, Math.log(1 + (this.N - df + 0.5) / (df + 0.5))); + } + + const qset = new Set(qterms); + const hits: SearchHit[] = []; + for (const d of this.docs) { + if (opts.pageType && (d.page.page_type ?? "content") !== opts.pageType) continue; + + let score = 0; + for (const t of qterms) { + const termIdf = idf.get(t) ?? 0; + const tf = d.tf.get(t); + if (tf) { + const denom = tf + K1 * (1 - B + B * (d.len / (this.avgdl || 1))); + score += termIdf * ((tf * (K1 + 1)) / denom); + } + // Field boosts: reward the term appearing in high-signal fields. + if (d.slugTok.has(t)) score += termIdf * W_SLUG; + if (d.titleTok.has(t)) score += termIdf * W_TITLE; + if (d.summaryTok.has(t)) score += termIdf * W_SUMMARY; + } + if (score > 0) { + score *= pageWeight(d.page.url); + hits.push({ + id: d.page.id, + title: d.page.title, + url: d.page.url, + summary: d.page.summary ?? "", + page_type: d.page.page_type ?? "content", + score: Number(score.toFixed(4)), + matching_section_ids: matchingSections(d.page, qset), + }); + } + } + hits.sort((a, b) => b.score - a.score); + return hits.slice(0, opts.limit ?? 10); + } +} diff --git a/build/docs-mcp-server/node/src/smoke.ts b/build/docs-mcp-server/node/src/smoke.ts new file mode 100644 index 0000000000..fedc6c450d --- /dev/null +++ b/build/docs-mcp-server/node/src/smoke.ts @@ -0,0 +1,82 @@ +// Offline smoke test: assertion-based checks of the index + tools + MCP +// response wrapping (no stdio transport). Exits non-zero on any failure. +// npm run smoke +// Runs against test/fixture.ndjson; the assertions encode fixture-specific +// ids/urls (incl. the colliding id="install" pair), so it is not meant to be +// pointed at the live feed. +import { fileURLToPath } from "node:url"; +import { loadFeed } from "./feed.js"; +import { DocsIndex } from "./search.js"; +import { searchDocs } from "./tools/search-docs.js"; +import { getPage } from "./tools/get-page.js"; +import { toolResult } from "./response.js"; + +let failures = 0; +function check(label: string, cond: boolean) { + console.log(`${cond ? "PASS" : "FAIL"} ${label}`); + if (!cond) failures++; +} + +const feed = + process.env.DOCS_NDJSON ?? + fileURLToPath(new URL("../test/fixture.ndjson", import.meta.url)); + +const pages = await loadFeed(feed); +const index = new DocsIndex(pages); +console.log(`loaded ${index.size} pages from ${feed}\n`); + +// --- search_docs --- +const stream = searchDocs(index, { query: "append an entry to a stream" }); +check("search returns hits", stream.count > 0); +check("search hits carry a url", Boolean(stream.results[0]?.url)); + +// --- get_page happy paths --- +const xadd = getPage(index, { id: "commands/xadd", roles: ["parameters"] }) as any; +check("get_page(unique id) resolves", xadd.id === "commands/xadd"); +check("roles filter returns only 'parameters'", (xadd.sections ?? []).every((s: any) => s.role === "parameters")); + +const exact = getPage(index, { url: "https://redis.io/docs/latest/operate/redisinsight/install/" }) as any; +check("get_page(exact url) resolves the right page", exact.title === "Install Redis Insight"); + +// exact url is authoritative even when a non-unique id is passed alongside it +// (Bugbot round-3 High): search hits carry both id + url, and id "install" is ambiguous. +const exactPlusId = getPage(index, { + url: "https://redis.io/docs/latest/operate/redisinsight/install/", + id: "install", +}) as any; +check("exact url + non-unique id resolves (not ambiguous)", exactPlusId.title === "Install Redis Insight"); + +const suffixUnique = getPage(index, { url: "/commands/xadd/" }) as any; +check("get_page(unambiguous partial url) resolves", suffixUnique.id === "commands/xadd"); + +// --- get_page ambiguity (Bugbot High + Codex Medium) --- +const ambId = getPage(index, { id: "install" }) as any; +check("ambiguous id returns error", typeof ambId.error === "string"); +check("ambiguous id lists candidates", (ambId.candidates ?? []).length === 2); + +const ambUrl = getPage(index, { url: "/install/" }) as any; +check("ambiguous partial url returns error (not silent first match)", typeof ambUrl.error === "string"); +check("ambiguous partial url lists candidates", (ambUrl.candidates ?? []).length === 2); + +// --- boundary-anchored suffix (Bugbot round-2 High): "add" must NOT match "xadd" --- +const boundary = getPage(index, { url: "add" }) as any; +check("partial url matches only on path-segment boundary (add !-> xadd)", typeof boundary.error === "string" && !boundary.candidates); + +// --- conflicting handles (Codex convergence model): url and id point at different pages --- +const conflict = getPage(index, { + url: "https://redis.io/docs/latest/develop/data-types/json/", + id: "commands/xadd", +}) as any; +check("conflicting url+id returns error", typeof conflict.error === "string"); +check("conflicting url+id lists both candidates", (conflict.candidates ?? []).length === 2); + +const missing = getPage(index, { id: "does-not-exist-anywhere" }) as any; +check("missing page returns error", typeof missing.error === "string"); + +// --- MCP response wrapping (Fix 2 / Bugbot Medium) --- +check("toolResult(missing) sets isError", toolResult(missing).isError === true); +check("toolResult(ambiguous id) sets isError", toolResult(ambId).isError === true); +check("toolResult(search) does NOT set isError", toolResult(stream).isError === undefined); + +console.log(`\n${failures === 0 ? "ALL PASSED" : failures + " FAILED"}`); +if (failures > 0) process.exit(1); diff --git a/build/docs-mcp-server/node/src/stem.ts b/build/docs-mcp-server/node/src/stem.ts new file mode 100644 index 0000000000..e94296f3b4 --- /dev/null +++ b/build/docs-mcp-server/node/src/stem.ts @@ -0,0 +1,136 @@ +// Compact Porter stemmer (classic algorithm). Applied to BOTH index and query +// tokens so word-form variants conflate (append/appends, prepend/prepending, +// expire/expires, queries/query). Consistency matters more than linguistic +// perfection here — the retrieval eval validates the net effect. + +const step2list: Record = { + ational: "ate", tional: "tion", enci: "ence", anci: "ance", izer: "ize", + bli: "ble", alli: "al", entli: "ent", eli: "e", ousli: "ous", + ization: "ize", ation: "ate", ator: "ate", alism: "al", iveness: "ive", + fulness: "ful", ousness: "ous", aliti: "al", iviti: "ive", biliti: "ble", + logi: "log", +}; +const step3list: Record = { + icate: "ic", ative: "", alize: "al", iciti: "ic", ical: "ic", ful: "", ness: "", +}; + +const c = "[^aeiou]"; +const v = "[aeiouy]"; +const C = c + "[^aeiouy]*"; +const V = v + "[aeiou]*"; +const mgr0 = "^(" + C + ")?" + V + C; +const meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; +const mgr1 = "^(" + C + ")?" + V + C + V + C; +const s_v = "^(" + C + ")?" + v; + +export function stem(w: string): string { + if (w.length < 3) return w; + + let stemmed: string; + let suffix: string; + let re: RegExp; + let re2: RegExp; + let re3: RegExp; + let re4: RegExp; + + const firstch = w.substr(0, 1); + if (firstch === "y") w = firstch.toUpperCase() + w.substr(1); + + // Step 1a + re = /^(.+?)(ss|i)es$/; + re2 = /^(.+?)([^s])s$/; + if (re.test(w)) w = w.replace(re, "$1$2"); + else if (re2.test(w)) w = w.replace(re2, "$1$2"); + + // Step 1b + re = /^(.+?)eed$/; + re2 = /^(.+?)(ed|ing)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + re = new RegExp(mgr0); + if (re.test(fp[1])) { + re = /.$/; + w = w.replace(re, ""); + } + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1]; + re2 = new RegExp(s_v); + if (re2.test(stemmed)) { + w = stemmed; + re2 = /(at|bl|iz)$/; + re3 = new RegExp("([^aeiouylsz])\\1$"); + re4 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re2.test(w)) w = w + "e"; + else if (re3.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } else if (re4.test(w)) w = w + "e"; + } + } + + // Step 1c + re = /^(.+?)y$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(s_v); + if (re.test(stemmed)) w = stemmed + "i"; + } + + // Step 2 + re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step2list[suffix]; + } + + // Step 3 + re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + suffix = fp[2]; + re = new RegExp(mgr0); + if (re.test(stemmed)) w = stemmed + step3list[suffix]; + } + + // Step 4 + re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/; + re2 = /^(.+?)(s|t)(ion)$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + if (re.test(stemmed)) w = stemmed; + } else if (re2.test(w)) { + const fp = re2.exec(w)!; + stemmed = fp[1] + fp[2]; + re2 = new RegExp(mgr1); + if (re2.test(stemmed)) w = stemmed; + } + + // Step 5a + re = /^(.+?)e$/; + if (re.test(w)) { + const fp = re.exec(w)!; + stemmed = fp[1]; + re = new RegExp(mgr1); + re2 = new RegExp(meq1); + re3 = new RegExp("^" + C + v + "[^aeiouwxy]$"); + if (re.test(stemmed) || (re2.test(stemmed) && !re3.test(stemmed))) w = stemmed; + } + + // Step 5b + re = /ll$/; + re2 = new RegExp(mgr1); + if (re.test(w) && re2.test(w)) { + re = /.$/; + w = w.replace(re, ""); + } + + return w.toLowerCase(); +} diff --git a/build/docs-mcp-server/node/src/tools/get-page.ts b/build/docs-mcp-server/node/src/tools/get-page.ts new file mode 100644 index 0000000000..d3326423b8 --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/get-page.ts @@ -0,0 +1,99 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; +import type { Page } from "../types.js"; + +export const GetPageInput = z + .object({ + id: z.string().optional(), + url: z.string().optional(), + roles: z.array(z.string()).optional(), + }) + .refine((v) => Boolean(v.id || v.url), { + message: "Provide either 'id' or 'url'.", + }); +export type GetPageInput = z.infer; + +/** + * The distinct pages (deduped by unique url) that the supplied handles resolve + * to. `url` is authoritative when it matches an exact page; otherwise it is + * treated as a path-boundary suffix. `id` is the last URL segment and is not + * unique, so it may contribute several pages. Collecting across all handles + * (rather than short-circuiting) means an ambiguous url still lets `id` help, + * and it surfaces the case where url and id point at *different* pages. + */ +function collectCandidates(index: DocsIndex, input: GetPageInput): Page[] { + const byUrl = new Map(); + const add = (p: Page | undefined) => { + if (p) byUrl.set(p.url, p); + }; + + // A boundary-suffix url (only reached when there was no exact match) plus id. + if (input.url) index.matchByUrlSuffix(input.url).forEach(add); + if (input.id) index.getPagesById(input.id).forEach(add); + return [...byUrl.values()]; +} + +function describeHandles(input: GetPageInput): string { + const parts: string[] = []; + if (input.url) parts.push(`url '${input.url}'`); + if (input.id) parts.push(`id '${input.id}'`); + return parts.join(" and "); +} + +function ambiguous(input: GetPageInput, candidates: Page[]) { + return { + error: `Ambiguous lookup: ${describeHandles(input)} matched ${candidates.length} pages. Call get_page again with a single, exact url.`, + candidates: candidates.map((p) => ({ title: p.title, url: p.url })), + }; +} + +function render(page: Page, input: GetPageInput) { + let sections = page.sections ?? []; + if (input.roles && input.roles.length) { + const want = new Set(input.roles.map((r) => r.toLowerCase())); + sections = sections.filter((s) => want.has((s.role ?? "").toLowerCase())); + } + return { + id: page.id, + title: page.title, + url: page.url, + summary: page.summary ?? "", + page_type: page.page_type ?? "content", + content_hash: page.content_hash, + sections, + }; +} + +/** Fetch one page, optionally filtered to sections with the given roles. */ +export function getPage(index: DocsIndex, input: GetPageInput) { + // An exact url is unique and authoritative. Return it directly rather than + // diluting it with a (possibly non-unique) id supplied alongside it — UNLESS + // the id points somewhere else entirely, which is a genuine conflict worth + // surfacing. A search hit's id is that page's own id, so the common + // exact-url + its-own-id case resolves cleanly. + if (input.url) { + const exact = index.getByUrl(input.url); + if (exact) { + if (input.id) { + const idPages = index.getPagesById(input.id); + if (idPages.length > 0 && !idPages.some((p) => p.url === exact.url)) { + const byUrl = new Map(); + [exact, ...idPages].forEach((p) => byUrl.set(p.url, p)); + return ambiguous(input, [...byUrl.values()]); + } + } + return render(exact, input); + } + } + + // Otherwise converge across the remaining handles (boundary-suffix url + id) + // and resolve only when they point at exactly one page. + const candidates = collectCandidates(index, input); + if (candidates.length === 0) { + return { error: `Page not found for ${describeHandles(input)}.` }; + } + if (candidates.length > 1) { + return ambiguous(input, candidates); + } + return render(candidates[0], input); +} diff --git a/build/docs-mcp-server/node/src/tools/search-docs.ts b/build/docs-mcp-server/node/src/tools/search-docs.ts new file mode 100644 index 0000000000..08f1574feb --- /dev/null +++ b/build/docs-mcp-server/node/src/tools/search-docs.ts @@ -0,0 +1,18 @@ +import { z } from "zod"; +import type { DocsIndex } from "../search.js"; + +export const SearchDocsInput = z.object({ + query: z.string().min(1, "query is required"), + page_type: z.enum(["content", "index"]).optional(), + limit: z.number().int().positive().max(50).optional(), +}); +export type SearchDocsInput = z.infer; + +/** Rank pages by relevance. Returns refs + summaries only — never full text. */ +export function searchDocs(index: DocsIndex, input: SearchDocsInput) { + const results = index.search(input.query, { + limit: input.limit ?? 10, + pageType: input.page_type, + }); + return { query: input.query, count: results.length, results }; +} diff --git a/build/docs-mcp-server/node/src/types.ts b/build/docs-mcp-server/node/src/types.ts new file mode 100644 index 0000000000..82ab8cc32d --- /dev/null +++ b/build/docs-mcp-server/node/src/types.ts @@ -0,0 +1,35 @@ +// Document schema as published in docs.ndjson / per-page index.json. +// See content/ai-agent-resources.md for the authoritative field reference. + +export interface Section { + id: string; + title: string; + /** Semantic role: overview | syntax | parameters | returns | example | ... */ + role?: string; + text: string; +} + +export interface Example { + id: string; + language: string; + code: string; + section_id: string; +} + +export interface Child { + title?: string; + url?: string; +} + +export interface Page { + id: string; + title: string; + url: string; + summary?: string; + /** "content" (has prose) or "index" (navigation only) */ + page_type?: string; + content_hash?: string; + sections?: Section[]; + examples?: Example[]; + children?: Child[]; +} diff --git a/build/docs-mcp-server/node/test/eval/cases.json b/build/docs-mcp-server/node/test/eval/cases.json new file mode 100644 index 0000000000..3e24aa50ce --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/cases.json @@ -0,0 +1,38 @@ +[ + { "kind": "command", "q": "append an entry to a stream", "expected": ["https://redis.io/docs/latest/commands/xadd/"] }, + { "kind": "command", "q": "add a member to a sorted set with a score", "expected": ["https://redis.io/docs/latest/commands/zadd/"] }, + { "kind": "command", "q": "set a string value only if the key does not already exist", "expected": ["https://redis.io/docs/latest/commands/setnx/", "https://redis.io/docs/latest/commands/set/"] }, + { "kind": "command", "q": "make a key expire after a given number of seconds", "expected": ["https://redis.io/docs/latest/commands/expire/", "https://redis.io/docs/latest/commands/pexpire/"] }, + { "kind": "command", "q": "atomically increment the integer stored at a key", "expected": ["https://redis.io/docs/latest/commands/incr/", "https://redis.io/docs/latest/commands/incrby/"] }, + { "kind": "command", "q": "remove a key from the database", "expected": ["https://redis.io/docs/latest/commands/del/", "https://redis.io/docs/latest/commands/unlink/"] }, + { "kind": "command", "q": "get all the fields and values stored in a hash", "expected": ["https://redis.io/docs/latest/commands/hgetall/"] }, + { "kind": "command", "q": "publish a message to a channel", "expected": ["https://redis.io/docs/latest/commands/publish/"] }, + { "kind": "command", "q": "listen for messages on a channel", "expected": ["https://redis.io/docs/latest/commands/subscribe/"] }, + { "kind": "command", "q": "prepend an element to the beginning of a list", "expected": ["https://redis.io/docs/latest/commands/lpush/"] }, + { "kind": "command", "q": "read a range of elements from a list", "expected": ["https://redis.io/docs/latest/commands/lrange/"] }, + { "kind": "command", "q": "check how long until a key expires", "expected": ["https://redis.io/docs/latest/commands/ttl/", "https://redis.io/docs/latest/commands/pttl/"] }, + { "kind": "command", "q": "add one or more members to a set", "expected": ["https://redis.io/docs/latest/commands/sadd/"] }, + { "kind": "command", "q": "run a server-side Lua script", "expected": ["https://redis.io/docs/latest/commands/eval/", "https://redis.io/docs/latest/commands/eval_ro/"] }, + { "kind": "command", "q": "retrieve the value of a string key", "expected": ["https://redis.io/docs/latest/commands/get/"] }, + { "kind": "command", "q": "incrementally iterate the keyspace without blocking the server", "expected": ["https://redis.io/docs/latest/commands/scan/"] }, + { "kind": "command", "q": "rename an existing key", "expected": ["https://redis.io/docs/latest/commands/rename/"] }, + { "kind": "command", "q": "set multiple fields on a hash at once", "expected": ["https://redis.io/docs/latest/commands/hset/", "https://redis.io/docs/latest/commands/hmset/"] }, + { "kind": "command", "q": "remove and return the first element of a list", "expected": ["https://redis.io/docs/latest/commands/lpop/", "https://redis.io/docs/latest/commands/blpop/"] }, + { "kind": "command", "q": "count the number of members in a set", "expected": ["https://redis.io/docs/latest/commands/scard/"] }, + { "kind": "command", "q": "store a JSON document at a path", "expected": ["https://redis.io/docs/latest/commands/json.set/"] }, + { "kind": "command", "q": "create a full-text search index", "expected": ["https://redis.io/docs/latest/commands/ft.create/"] }, + + { "kind": "concept", "q": "connect to Redis from a Python application", "expected": ["https://redis.io/docs/latest/develop/clients/redis-py/connect/"] }, + { "kind": "concept", "q": "connect to Redis using the Jedis Java client", "expected": ["https://redis.io/docs/latest/develop/clients/jedis/connect/"] }, + { "kind": "concept", "q": "what are the differences between the Redis data types", "expected": ["https://redis.io/docs/latest/develop/data-types/compare-data-types/", "https://redis.io/docs/latest/develop/data-types/"] }, + { "kind": "concept", "q": "how do Redis transactions work", "expected": ["https://redis.io/docs/latest/develop/using-commands/transactions/"] }, + { "kind": "concept", "q": "send several commands together to reduce round trips", "expected": ["https://redis.io/docs/latest/develop/using-commands/pipelining/"] }, + { "kind": "concept", "q": "configure persistence with RDB snapshots and the append-only file", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/"] }, + { "kind": "concept", "q": "set up replication between a primary and its replicas", "expected": ["https://redis.io/docs/latest/operate/oss_and_stack/management/replication/"] }, + { "kind": "concept", "q": "run a vector similarity search over embeddings", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/query/vector-search/", "https://redis.io/docs/latest/develop/ai/search-and-query/"] }, + { "kind": "concept", "q": "get notified when keys are changed or expire", "expected": ["https://redis.io/docs/latest/develop/pubsub/keyspace-notifications/"] }, + { "kind": "concept", "q": "cache data on the client side to reduce load", "expected": ["https://redis.io/docs/latest/develop/clients/client-side-caching/", "https://redis.io/docs/latest/develop/reference/client-side-caching/"] }, + { "kind": "concept", "q": "work with the string data type", "expected": ["https://redis.io/docs/latest/develop/data-types/strings/"] }, + { "kind": "concept", "q": "use hashes to store object-like records", "expected": ["https://redis.io/docs/latest/develop/data-types/hashes/"] }, + { "kind": "concept", "q": "full-text search and secondary indexing over Redis data", "expected": ["https://redis.io/docs/latest/develop/ai/search-and-query/"] } +] diff --git a/build/docs-mcp-server/node/test/eval/run.mjs b/build/docs-mcp-server/node/test/eval/run.mjs new file mode 100644 index 0000000000..6c8da79419 --- /dev/null +++ b/build/docs-mcp-server/node/test/eval/run.mjs @@ -0,0 +1,84 @@ +// AI-answer eval (retrieval quality) for the docs MCP server. +// Runs each question in cases.json through search_docs and measures whether the +// expected canonical page is retrieved (recall@k, MRR). A data-integrity check +// flags any expected url that isn't in the feed, so a bad ground-truth entry is +// reported rather than silently scored as a miss. +// +// node test/eval/run.mjs # uses cached test/eval/docs.ndjson.gz +// DOCS_NDJSON= node test/eval/run.mjs +// +// Imports the BUILT server (dist/), so run `npm run build` first. +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { loadFeed } from "../../dist/feed.js"; +import { DocsIndex } from "../../dist/search.js"; +import { searchDocs } from "../../dist/tools/search-docs.js"; + +const K = [1, 3, 5, 10]; +const LIMIT = 10; +const norm = (u) => u.trim().toLowerCase().replace(/\/+$/, ""); +const short = (u) => (u ? u.replace("https://redis.io/docs/latest", "") : "—"); + +const feedSrc = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./docs.ndjson.gz", import.meta.url)); +const cases = JSON.parse(await readFile(fileURLToPath(new URL("./cases.json", import.meta.url)), "utf8")); + +const pages = await loadFeed(feedSrc); +const index = new DocsIndex(pages); +const feedUrls = new Set(pages.map((p) => norm(p.url))); + +const broken = []; +const rows = []; +for (const c of cases) { + const expected = c.expected.map(norm); + if (expected.every((u) => !feedUrls.has(u))) { + broken.push({ q: c.q, missing: expected }); + continue; + } + const results = searchDocs(index, { query: c.q, limit: LIMIT }).results.map((r) => norm(r.url)); + let rank = null; + for (let i = 0; i < results.length; i++) { + if (expected.includes(results[i])) { + rank = i + 1; + break; + } + } + rows.push({ kind: c.kind ?? "command", q: c.q, rank, top: results[0] }); +} + +function metrics(set) { + const n = set.length || 1; + const recall = Object.fromEntries( + K.map((k) => [k, set.filter((r) => r.rank && r.rank <= k).length / n]), + ); + const mrr = set.reduce((s, r) => s + (r.rank ? 1 / r.rank : 0), 0) / n; + return { recall, mrr }; +} + +console.log( + `Feed: ${pages.length} pages | cases scored: ${rows.length}` + + (broken.length ? ` | ${broken.length} BROKEN (expected url not in feed)` : "") + + "\n", +); +for (const r of rows) { + const tag = r.rank ? `#${r.rank}`.padEnd(5) : "MISS "; + console.log(`${tag} [${r.kind.slice(0, 4)}] ${r.q}${r.rank ? "" : ` [rank-1 was: ${short(r.top)}]`}`); +} + +const groups = [ + ["overall", rows], + ["command", rows.filter((r) => r.kind === "command")], + ["concept", rows.filter((r) => r.kind === "concept")], +]; +console.log("\n--- retrieval quality (recall@1 / @3 / @5 / @10 | MRR) ---"); +for (const [label, set] of groups) { + if (!set.length) continue; + const m = metrics(set); + const cells = K.map((k) => `${(m.recall[k] * 100).toFixed(0)}%`.padStart(4)).join(" / "); + console.log(`${label.padEnd(8)} (n=${String(set.length).padStart(2)}): ${cells} | ${m.mrr.toFixed(3)}`); +} + +if (broken.length) { + console.log("\n--- BROKEN eval cases (fix ground truth) ---"); + for (const b of broken) console.log(` "${b.q}" -> not in feed: ${b.missing.map(short).join(", ")}`); +} diff --git a/build/docs-mcp-server/node/test/fixture.ndjson b/build/docs-mcp-server/node/test/fixture.ndjson new file mode 100644 index 0000000000..95b767477c --- /dev/null +++ b/build/docs-mcp-server/node/test/fixture.ndjson @@ -0,0 +1,5 @@ +{"id":"commands/xadd","title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/","summary":"Appends a new entry to a stream.","page_type":"content","content_hash":"abc123","sections":[{"id":"overview","title":"XADD","role":"overview","text":"Appends the specified stream entry to the stream at the specified key. If the key does not exist, a new stream is created."},{"id":"parameters","title":"Parameters","role":"parameters","text":"key: the name of the stream. NOMKSTREAM: optional, do not create the stream if it does not exist. field value: one or more field-value pairs to add to the stream entry."},{"id":"return","title":"Return value","role":"returns","text":"Returns the ID of the added stream entry."}],"examples":[{"id":"overview-ex0","language":"python","code":"r.xadd('mystream', {'field': 'value'})","section_id":"overview"}],"children":[]} +{"id":"develop/data-types/json","title":"JSON","url":"https://redis.io/docs/latest/develop/data-types/json/","summary":"Store and query JSON documents in Redis.","page_type":"content","content_hash":"def456","sections":[{"id":"overview","title":"Redis JSON","role":"overview","text":"The JSON data type lets you store, update, and retrieve JSON values in a Redis database."},{"id":"example","title":"Example","role":"example","text":"Use JSON.SET to store a document and JSON.GET to retrieve values from it by path."}],"examples":[{"id":"example-ex0","language":"python","code":"r.json().set('doc', '$', {'a': 1})","section_id":"example"}],"children":[]} +{"id":"commands","title":"Commands","url":"https://redis.io/docs/latest/commands/","summary":"Reference documentation for all Redis commands.","page_type":"index","children":[{"title":"XADD","url":"https://redis.io/docs/latest/commands/xadd/"}]} +{"id":"install","title":"Install Redis","url":"https://redis.io/docs/latest/operate/oss_and_stack/install/","summary":"Install Redis Open Source on your platform.","page_type":"content","content_hash":"aaa111","sections":[{"id":"overview","title":"Install Redis","role":"overview","text":"Install Redis Open Source on Linux, macOS, or Windows."}],"examples":[],"children":[]} +{"id":"install","title":"Install Redis Insight","url":"https://redis.io/docs/latest/operate/redisinsight/install/","summary":"Install the Redis Insight GUI.","page_type":"content","content_hash":"bbb222","sections":[{"id":"overview","title":"Install Redis Insight","role":"overview","text":"Download and install the Redis Insight desktop application."}],"examples":[],"children":[]} diff --git a/build/docs-mcp-server/node/test/mcp-client.mjs b/build/docs-mcp-server/node/test/mcp-client.mjs new file mode 100644 index 0000000000..5010723f0e --- /dev/null +++ b/build/docs-mcp-server/node/test/mcp-client.mjs @@ -0,0 +1,44 @@ +// Manual local test: drive the built stdio server through the real MCP client +// (spawn -> initialize -> tools/list -> tools/call), against the local fixture. +// node test/mcp-client.mjs +import { Client } from "@modelcontextprotocol/sdk/client/index.js"; +import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"; +import { fileURLToPath } from "node:url"; + +const feed = + process.env.DOCS_NDJSON ?? fileURLToPath(new URL("./fixture.ndjson", import.meta.url)); +const serverEntry = fileURLToPath(new URL("../dist/index.js", import.meta.url)); + +const transport = new StdioClientTransport({ + command: "node", + args: [serverEntry], + env: { ...process.env, DOCS_NDJSON: feed }, +}); + +const client = new Client({ name: "local-test-client", version: "0.0.1" }, { capabilities: {} }); +await client.connect(transport); +console.log("connected to server\n"); + +const tools = await client.listTools(); +console.log("tools/list ->", tools.tools.map((t) => t.name).join(", "), "\n"); + +const search = await client.callTool({ + name: "search_docs", + arguments: { query: "append an entry to a stream" }, +}); +console.log("tools/call search_docs('append an entry to a stream'):"); +console.log(search.content[0].text, "\n"); + +const amb = await client.callTool({ name: "get_page", arguments: { id: "install" } }); +console.log(`tools/call get_page(id='install') -> isError=${amb.isError}`); +console.log(amb.content[0].text, "\n"); + +const page = await client.callTool({ + name: "get_page", + arguments: { url: "https://redis.io/docs/latest/commands/xadd/", roles: ["parameters"] }, +}); +console.log(`tools/call get_page(url=.../commands/xadd/, roles=['parameters']) -> isError=${page.isError}`); +console.log(page.content[0].text); + +await client.close(); +console.log("\nclosed cleanly"); diff --git a/build/docs-mcp-server/node/tsconfig.json b/build/docs-mcp-server/node/tsconfig.json new file mode 100644 index 0000000000..01b2904d2f --- /dev/null +++ b/build/docs-mcp-server/node/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "esnext", + "lib": ["ES2022"], + "outDir": "./dist", + "rootDir": "./src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "forceConsistentCasingInFileNames": true, + "resolveJsonModule": true, + "declaration": true, + "sourceMap": true, + "moduleResolution": "node" + }, + "include": ["src/**/*"], + "exclude": ["node_modules", "dist", "src/smoke.ts"] +}