Rebuild runbook registry with hybrid retrieval; hard cutover to task-typed embedding

[Savid]

Summary

Three layers, bottom to top: the proxy embedding API becomes versioned — the existing v1/v2 routes are preserved byte-compatible and a new task-typed v3 is added; semantic search over runbooks becomes hybrid and multi-vector on top of it; and the runbook registry is rebuilt against an explicit authoring standard with retrieval-quality regression checks.

Note: an earlier revision of this branch did a hard cutover (single unversioned /embedding, v1/v2 deleted). That was abandoned — deployed fleets can't flag-day. The current design keeps old binaries working unchanged.

Embedding: versioned endpoints

Route	Contract	Config	Cache key
/embed (+/check) — v1	symmetric, no dims	embedding:	{model}:{hash}
/v2/embedding (+/check) — v2	symmetric, fixed dims, model+dims echoed	embedding_v2:	{model}:{dims}:{hash}
/v3/embedding (+/check) — v3	task=query|document required → upstream input_type (asymmetric retrieval embedding); 400 on missing/unknown task	shares embedding_v2:	{model}:{dims}:{task}:{hash}

• v3 is a protocol upgrade, not a new embedding space — it shares v2's service, config, and Redis namespace; key shapes are provably disjoint, and warm v1/v2 cache entries survive the proxy upgrade.
• Servers negotiate newest-first: v3 probe → v2 probe (a dims-less echo from a historical proxy is tolerated) → confirmed v1 fallback via the restored embedding_model datasources field.
• The search runtime treats (model, dimensions, protocol) as the index identity and auto-re-indexes when any changes — a proxy upgrading v2→v3 under a running server flips into task-typed retrieval on the next discovery tick, no restart. Clients verify the advertised space on every v2/v3 response and fail builds fast rather than caching wrong-space vectors.
• Hardening: upstream vector-length validation against configured dims, dimensions ≥ 1 validation, bounded items and hashes on every route.

Deploy notes (no ordering requirement): old servers keep working against the new proxy on v1/v2 unchanged; new servers against an old proxy negotiate v2 (symmetric search, one log line noting task-typed retrieval awaits v3) and self-upgrade when the proxy does. Existing embedding:/embedding_v2: config blocks load with identical semantics; a dimensions: under the v1 block logs a warning that it's ignored.

Retrieval

• pkg/textmatch: lexical token/trigram scorer shared between the resource resolver and search.
• The runbook index is hybrid and multi-vector: a document-space metadata vector; one vector per trigger embedded in query space (a trigger is a hypothetical caller query — same-space embedding makes near-verbatim matches score ~1.0 on v3); per-section child vectors discounted ×0.95 so the authored routing surface wins near-ties. Score = best dense + 0.3 × lexical, unclamped, deterministically ordered. On v1/v2 proxies the same index works symmetric.
• dotProduct guards mismatched vector lengths; a once-per-index warning covers the dims-mismatch window.

Runbook registry

• Rebuilt as single-responsibility runbooks split by co-retrieval, with required triggers frontmatter (the primary retrieval surface), one-canonical-owner fact discipline, filled-example output contracts, and repo-wide ref-integrity tests. runbooks/AGENTS.md documents the standard; root AGENTS.md is the real guide with CLAUDE.md as a shim.
• scripts/runbook-retrieval-check.sh: a golden-query matrix (51 queries incl. confusable near-miss pairs and out-of-scope probes) asserting top-1 retrieval against a running server — currently 51/51.

Verification

• Full test suite with race detector + lint: clean. Independent review passes (wire-compat vs deployed shapes, state/concurrency, validation/tests, adversarial) with all findings fixed and re-verified.
• Live end-to-end, three phases: fresh v3 negotiation and task-typed index build; a live downgrade to the actual pre-cutover proxy binary (v2 negotiated on the refresh tick, dims-less echo tolerated, symmetric index rebuilt through the old binary); a live upgrade back (protocol flip detected, task-typed re-index from warm cache in ~2s). Zero errors across all phases.

🤖 Generated with Claude Code

[github-actions]

🐼 Smoke eval — 2b5276d: ❌ 0/8 pass

📊 Interactive report — tokens p50 0 · tokens/solve 0.

Reference points: feat/runbook-registry-embedding-cutover@ee6c40f 0% · feat/runbook-registry-embedding-cutover@78489e6 0% · feat/runbook-registry-embedding-cutover@6ebd2e1 0%.

question	result	tokens	tools
forky_node_coverage	❌	14,631	4
tracoor_node_coverage	❌	14,102	4
mainnet_block_arrival_p50	❌	16,177	8
list_datasources	❌	13,457	3
block_count_24h	❌	16,571	10
missed_slots_24h	❌	14,721	5
chartkit_default_arrival_distribution	❌	39,064	14
storage_upload_session_scoped	❌	30,628	15

🔭 Langfuse traces (8 runs; ⚠️ = failed)

• forky_node_coverage ⚠️
• tracoor_node_coverage ⚠️
• mainnet_block_arrival_p50 ⚠️
• list_datasources ⚠️
• block_count_24h ⚠️
• missed_slots_24h ⚠️
• chartkit_default_arrival_distribution ⚠️
• storage_upload_session_scoped ⚠️

_{The report walks this branch's commits against the master baseline and the most recent release. A self-contained copy is in the run's eval-smoke-* artifact.}