Architecture hardening: conservative rights merge, unified pagination cursor, CJK ranking, bounded fan-out, leaner MCP#17
Merged
Conversation
- merge: resolve cross-source license conflicts conservatively (stricter license wins; incomparable claims collapse to unknown), surfaced in meta.warnings and via merge.onRightsConflict - client: unified load-more cursor (SearchInput.cursor + meta.nextCursor) with cross-page dedup; cache keys embed the full normalized query (no more 32-bit hash collisions); new cacheRaw option; runProvider extracted to provider-run.ts - query: single-track capability routing — queryFeatures/filters deprecated, capabilities.controls is the only routing source, and the legacy NormalizedQuery.filters channel is derived from routed controls - rerank: CJK-aware tokenize (character bigrams) so CJK queries score - mcp: BYOK providers become lazily-loaded optionalDependencies; search_references gains rerank + cursor - providers: wire controls.page in openverse, pixabay, internet-archive, wikimedia-commons (gsroffset), artic Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
- providers: wire controls.page across every source with usable pagination (unsplash/flickr/freesound native page; smithsonian/ europeana start; jamendo/ambientcg offset; met/polyhaven windowed slice) — the unified cursor now advances every paginatable source - core: RefkitOptions.concurrency bounds in-flight provider searches (default unlimited; queued providers keep their full timeout) - tooling: eslint (typescript-eslint syntactic recommended) + pnpm lint + CI lint step; fix the four existing violations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
- mcp/tsup: externalize @refkit/* so optionalDependencies providers are imported from node_modules at runtime instead of being bundled into dist (--omit=optional now works; provider patches reach users) - cursor: drain the overfetched pool of the current provider page before advancing (no more skipped ranked results); advance internally up to 3 pages when a page is fully seen/gated; cap the seen set; validate the cursor with zod; bind fixes with regression tests - mcp: return nextCursor unconditionally at the top level of search_references output — pagination no longer requires explain - mcp/cli: classify BYOK load failures (ERR_MODULE_NOT_FOUND vs real errors, real error surfaced) and load sources concurrently - merge: report each cross-source rights conflict once per URL with only source-declared licenses (no phantom 'unknown' claims, no duplicates); RightsConflict.licenses is now LicenseId[] - cache: fixed-shape hashed keys (strict-KV-safe) + query fingerprint verified on read, so a key collision degrades to a miss - createRefkit: reject non-array providers (un-awaited async factory) with a clear error at construction time - query: capabilities-less providers declaring legacy queryFeatures keep receiving their filters (loud deprecation, not silent degradation) - internet-archive: escape Lucene syntax in the user query - provider-helpers: shared offsetForPage; providers use validated setIfPositiveInt/offsetForPage for all page wiring; drop the dead useLegacyFilter fallbacks Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
Verified the three release-blocking findings A/B against the pre-fix commit (a88f105) with runtime repros: overfetch gap (20/60 vs 60/60 results returned), false exhaustion (page-3 unreachable vs reached), and optional-provider bundling (9 bundled chunks/220K vs 0/80K with imports resolving from node_modules); built cli answers the MCP initialize handshake with a BYOK key set. This adds the missing unit test for top-level nextCursor without explain + cursor round-trip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
- scripts/smoke-artifact.mjs: pack every package (publishConfig applied, like a real publish), install the tarballs into a clean project, and boot the installed @refkit/mcp cli through three scenarios — keyless handshake, BYOK dynamic import from node_modules, and graceful skip when the optional package is removed. Guards the layer source suites cannot see (a bundler regression re-inlining optionalDependencies fails scenario 3). Wired into CI after build as `pnpm smoke:artifact`. - mcp: BYOK_SOURCES exported + test pinning it 1:1 to package.json optionalDependencies (and out of hard dependencies) - core: offsetForPage unit tests (offsets, 1-based origin, rejection of page 1 / non-integer / negative input) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the findings of an architecture review of core/providers/mcp, then a follow-up code review of the branch itself (10 findings, all fixed, the 3 release-blocking ones verified A/B against the pre-fix commit).
Core (
@refkit/core, minor)unknown(→ needs-review). Conflicts are reported once per URL viameta.warningsand the newmerge.onRightsConflictobserver (RightsConflict,stricterLicenseexported). Previously whichever provider was registered first silently won — fail-open by accident.SearchInput.cursor+meta.nextCursor: drains the current provider page's overfetched pool before advancing the provider-local page internally (up to 3 pages per call), dedupes against everything already returned (seen-set capped at 500), and validates cursors with zod. Verified by runtime repro: pre-fix load-more returned 20/60 results and terminated early on a duplicate page; now 60/60 with page-3 reachability.refkit:v2:.queryFeatures/filtersdeprecated; routing driven bycapabilities.controls, with the deprecatedNormalizedQuery.filterschannel derived from routed controls so both channels always agree. Capabilities-less providers declaring legacyqueryFeatureskeep receiving their filters (loud deprecation, not silent degradation).tokenize— CJK runs tokenize into character bigrams solexicalRerankerscores Chinese/Japanese/Korean queries instead of dropping them.concurrencyoption (default unlimited); a queued provider's timeout starts only when its slot starts.cacheRaw: falseoption;runProviderSearch/providerCacheKey/stableStringify/offsetForPageextracted and exported;createRefkitrejects non-arrayproviders(un-awaited async factory) at construction.MCP (
@refkit/mcp, minor)search_referencesgainsrerank: true(CJK-aware lexical reranking) andcursor; the continuation token is returned as top-levelnextCursoron every call, independent ofexplain.optionalDependencies, loaded lazily and concurrently only when their key is set; load failures are classified (ERR_MODULE_NOT_FOUND→ "not installed" hint, anything else surfaces the real error).defaultProviders()is now async.@refkit/*— pre-fix builds bundled all 9 optional providers intodist/cli.js(220K,--omit=optionalineffective, provider patches snapshotted at publish time); verified post-fix dist resolves them from node_modules (80K, real package imports) and the built cli answers the MCP initialize handshake.Providers (13 × patch)
controls.pagewired through shared validated helpers (setIfPositiveInt/offsetForPage): native page params (openverse, pixabay, unsplash, flickr, freesound, gutendex, pexels, artic, internet-archive), offset translation (wikimediagsroffset, smithsonian/europeanastart, jamendo/ambientCGoffset), windowed slices (met, polyhaven). Brave/PoetryDB/Rijksmuseum expose no usable offset pagination and keeppageundeclared.)broke the expression;) OR (mediatype:...could escape the movies/texts scope).useLegacyFilterfallbacks removed from pexels/pixabay/unsplash.Tooling
pnpm lint+ CI lint step; 4 existing violations fixed.Verification
pnpm typecheck(0 errors) ·pnpm lint·pnpm test:run(357 passed, live suites skipped) ·pnpm build· A/B repros for the three release-blocking findings against pre-fix commit a88f105 · built MCP cli boot + initialize handshake with a BYOK key set. Changesets included (core/mcp minor, 13 provider patches).🤖 Generated with Claude Code
https://claude.ai/code/session_018hoN6wKWxcs8WbUNCM9rtt
Generated by Claude Code