DOC-6809 Docs-only MCP server: spec + v0 prototype#3585
Open
andy-stark-redis wants to merge 9 commits into
Open
DOC-6809 Docs-only MCP server: spec + v0 prototype#3585andy-stark-redis wants to merge 9 commits into
andy-stark-redis wants to merge 9 commits into
Conversation
Investigation into a read-only "docs MCP server" — distinct from the existing data-plane redis/mcp-redis — that exposes the documentation corpus as agent-queryable tools over the docs.ndjson feed we already publish. Spec plus a working v0 (search_docs + get_page, self-contained BM25, no datastore). Two findings from running it against the live feed shaped the design: indexing 2,531 pages takes ~0.3s and queries ~100ms, so search compute is a non-issue and Rust/WASM would be premature optimisation; but untuned lexical ranking mis-ranks natural-language queries — with no stemmer, "append" misses XADD's "appends", and canonical command pages lose to release-notes, operator custom-resources, and client-library pages that repeat the same terms. That ranking gap, not speed, is what justifies the vector-search upgrade path in the spec. Learned: measured perf rules out WASM; lexical ranking (not speed) is the real limit Constraint: docs MCP server stays read-only, no DB connection, no creds (unlike mcp-redis data plane) Rejected: Rust/WASM for search speed | ~0.3s index + ~100ms queries on 2,531 pages, compute isn't the bottleneck Directive: don't over-tune lexical BM25 weights (overfits to sample queries); fix ranking via stemming/analyzer or vector search Reversibility: clean Ticket: DOC-6809 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
Contributor
🛡️ Jit Security Scan Results
✅ No security findings were detected in this PR
Security scan by Jit
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… Codex) Address Cursor Bugbot findings on PR #3585 plus a Codex follow-up review. The feed's `id` is the last URL path segment and is NOT unique (~213 ids map to several pages), so `byId` is now a multimap and `get_page` resolves by the unique `url` first; an ambiguous `id` or partial `url` returns candidate urls instead of silently returning the wrong page. The MCP handler now sets `isError` when a tool result carries an `error`. Smoke test rewritten as assertions (incl. a colliding id="install" fixture pair) so this can't regress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test/mcp-client.mjs drives the built stdio server through the real MCP client (spawn -> initialize -> tools/list -> tools/call) against the fixture, so the transport path and isError signalling are exercised end-to-end, not just the tool logic. Live-feed testing surfaced that the spec's assumed section roles (syntax/examples) don't match the feed (it uses content/parameters/example/ returns, no syntax), so a roles filter can silently return nothing — recorded in SPEC open questions to resolve before building get_examples/get_command. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-2 Bugbot findings on the get_page/search resolution logic were churn: the round-1 duplicate-id fix introduced two of them. Rather than three point- patches, consolidate resolution into one convergence model (per Codex design review): collect candidate pages across all supplied handles (exact url, id, boundary-anchored suffix url) and resolve only when they converge on exactly one page — so an ambiguous url still tries id, and url/id pointing at different pages reports conflicting handles instead of silently applying precedence. - Partial-url suffix matching is now anchored to "/" path segments, so "get" matches .../commands/get but not .../config-get or .../arget (was 19 false matches on the live feed, now 2 legitimate ones). [Bugbot High] - Ambiguous url no longer short-circuits the id fallback. [Bugbot Medium] - Removed the `version` param: it advertised a `latest` default it never enforced. Deferred until a committed URL/version model exists. [Bugbot Medium] - Smoke test extended with boundary and conflicting-handles assertions (17 total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…icient npm run eval scores search_docs retrieval against a curated question set (test/eval/cases.json — command lookups phrased without the command name), reporting recall@k / MRR with a data-integrity check that flags expected urls missing from the feed. Feed read from DOCS_NDJSON or a gitignored local cache. Baseline on the live feed (22 cases): recall@1 32%, @3 45%, @5 59%, @10 73%, MRR 0.42. Confirms with data what was hypothesised: lexical retrieval alone is not good enough — canonical command pages lose to sibling commands (zadd -> zincrby), operator pages (del -> remove-node), and concept pages (ft.create -> a full-text concept page), and there is no stemming (append !-> appends). This is the measured case for the stemming/boost/vector work in SPEC §6/§10 and lets future ranking changes be compared against a baseline instead of tuned blind. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two principled ranking changes, measured against the eval baseline: - Porter stemmer (src/stem.ts) applied to both index and query tokens so word forms conflate (append/appends, prepend/prepending, expires/expire). Stopwords are filtered on raw tokens before stemming. - Page-type weighting on the final score: demote release-notes/REST-API (x0.5) and operator (x0.7) pages, modestly boost /commands/* (x1.5) — targeting the observed failures where operator/concept pages outranked command pages. Eval (22 command-lookup cases): recall@1 32->50, @3 45->68, @5 59->86, @10 73->95, MRR 0.42->0.65; misses 6->1. Caveat recorded in README/SPEC: the eval is command-heavy so the /commands/* boost partly flatters it — concept/ how-to cases still needed before calling lexical sufficient generally, and the residual miss (del/unlink beaten by flushdb for "remove a key") is a semantic gap that motivates vector search. Smoke still green (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bugbot High: collectCandidates always merged id matches even when url already resolved to an exact page, so get_page(url=<exact>, id=<its non-unique id>) — the natural flow when an agent echoes both fields from a search hit — returned an ambiguous error instead of the page. The consolidation's "always converge" rule over-corrected: an exact url is unique and should win. Fix: an exact url short-circuits to that page, EXCEPT when a supplied id points somewhere else entirely (idPages don't include the exact page) — still a genuine conflict, so still reported. This reconciles the Bugbot finding with Codex's earlier conflict-detection ask: exact-url + its-own-id resolves; exact-url + a different id conflicts. Smoke covers both (17 assertions). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit eb0e7ca. Configure here.
| const pages = await loadFeed(FEED_SOURCE); | ||
| const index = new DocsIndex(pages); | ||
| // Log to stderr — stdout is reserved for the MCP protocol. | ||
| console.error(`[redis-docs-mcp] indexed ${index.size} pages from ${FEED_SOURCE}`); |
There was a problem hiding this comment.
Server starts with empty index
Medium Severity
After loadFeed returns no pages (empty file, bad path, or no valid NDJSON lines), main still builds a DocsIndex and advertises readiness. Tools then always return empty search results or get_page not-found errors with no fatal startup failure.
Reviewed by Cursor Bugbot for commit eb0e7ca. Configure here.
De-bias the retrieval eval (was 22 command-only cases): tag each case command/concept and add 13 concept/how-to cases with feed-verified ground truth (incl. pages under /operate/, to test the demotion). Runner now reports recall per kind. Result under the shipped weighting: command recall@5 86% / MRR 0.65 but **concept recall@5 46% / MRR 0.29** — concept retrieval is ~half as good. The /commands/* x1.5 boost is the cause: it ranks command pages above the canonical concept page when both compete (persistence -> bgrewriteaof, replication -> cluster-replicate, keyspace-notifications -> expire), and the blanket /operate/ demotion drags down legitimate concept pages. Ablation (neutralise command boost, demote only rest-api/release-notes/references): concept @5 46->62, MRR .29->.45; command @5 86->73. Command-optimised vs balanced is a workload call — left as an open decision in SPEC, weighting unchanged for now. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the concept-case eval finding, switch page weighting from command-optimised to balanced: demote only release-notes/REST-API/references (x0.5); drop the /commands/* x1.5 boost and the blanket /operate/ x0.7 demotion. This trades command recall@5 86->73 for concept 46->62 and the best overall MRR (0.53), and stops burying legitimate /operate/ concept pages (persistence, replication). Command ranking should be lifted by better signal (vectors), not the boost. Eval + smoke green; SPEC decision marked resolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: not finished - just using the PR for Bugbot feedback so far.
What
Investigation (DOC-6809) into a read-only docs MCP server — a knowledge-plane server that exposes the Redis documentation corpus as agent-queryable tools, distinct from the existing data-plane
redis/mcp-redis(which reads/writes a live Redis instance). It's read-only, needs no database or credentials, and returns citations to our docs.It reuses the feed we already publish (Hugo per-page
index.json→generate_ndjson.py→docs.ndjson), so there's no new content pipeline.Contents
build/docs-mcp-server/SPEC.md— design: 5-tool surface (search_docs,get_page,get_section,get_examples,get_command), deployment/search-backend matrix, phased plan.build/docs-mcp-server/node/— a working v0 prototype (TypeScript,@modelcontextprotocol/sdk, stdio) with a self-contained BM25 index and two tools:search_docs+get_page. Includes an offline smoke test (npm run smoke).Findings from running v0 against the live feed (2,531 pages)
Status
Investigation / prototype. Not wired into any published site.
search_docs/get_pageonly;get_examples,get_section,get_commandare v1.🤖 Generated with Claude Code
Note
Low Risk
New build-only prototype; read-only public docs, no credentials or site integration, and no changes to the published docs build pipeline.
Overview
Adds a read-only docs MCP server investigation: a design spec plus a working v0 stdio prototype under
build/docs-mcp-server/that queries the existingdocs.ndjsonfeed (no new Hugo pipeline).SPEC.mddefines a knowledge-plane MCP (distinct from data-planemcp-redis): five planned tools, stdio vs hosted deployment, lexical v1 vs vector v2, and phased delivery. v0 implements onlysearch_docsandget_page, with version filtering explicitly deferred.node/is a TypeScript MCP server (@modelcontextprotocol/sdk, Zod) that loads local or remote NDJSON (gzip-aware viaDOCS_NDJSON), builds an in-memory BM25 index with Porter stemming, field boosts, and balanced URL demotion (release-notes / REST API / references). Search returns refs only;get_pagesupports role-filtered sections, treats URL as authoritative over non-uniqueid, and returns structured errors for ambiguous or conflicting lookups. MCP responses setisErrorwhen tools return{ error: ... }.Also ships offline smoke tests, a 35-case retrieval eval (
npm run eval), fixture NDJSON, and a manual MCP client script—documented in README with measured recall metrics.Reviewed by Cursor Bugbot for commit cf2f900. Bugbot is set up for automated code reviews on this repo. Configure here.