Architecture hardening: conservative rights merge, unified pagination cursor, CJK ranking, bounded fan-out, leaner MCP

.changeset/architecture-review-core.md

@@ -0,0 +1,14 @@
+---
+"@refkit/core": minor
+---
+
+Architecture-review hardening:
+
+- **Conservative cross-source rights merge** — when two sources disagree about the license of the same canonical URL, the stricter license wins (incomparable claims collapse to `unknown` → needs-review); conflicts surface in `meta.warnings` and via the new `merge.onRightsConflict` observer. New export: `stricterLicense`, `RightsConflict`.
+- **Unified pagination cursor** — `SearchInput.cursor` + `meta.nextCursor`: opaque load-more cursor that first drains the current provider page's overfetched pool, then advances the provider-local page internally, deduping against previously returned results (seen-set capped so cursors stay small).
+- **CJK-aware `tokenize`** — CJK runs tokenize into character bigrams, so `lexicalReranker` scores Chinese/Japanese/Korean queries instead of dropping them.
+- **Collision-proof cache keys** — per-provider cache keys stay short and fixed-shape (safe for strict KV backends), while the cached value embeds the full normalized-query fingerprint and is verified on read: a key collision degrades to a cache miss, never to another query's results. Existing cache entries are invalidated by the format change (`refkit:v2:`).
+- New `cacheRaw: false` option strips `raw` provider payloads from cache entries.
+- New `concurrency` option bounds how many provider searches run at once per search call (default unlimited, matching previous behavior); a queued provider's timeout starts only when it actually runs.
+- **Deprecations (single-track capability routing)** — `SearchFilters`, `SearchInput.filters`, `NormalizedQuery.filters`, `ReferenceProvider.queryFeatures` (now optional), and `QueryFeature` are deprecated. Routing is driven solely by `capabilities.controls`; legacy `filters` are merged into `controls` and the deprecated `NormalizedQuery.filters` channel is derived from the routed controls, so both channels always agree. Providers that declared filter support only via `queryFeatures` must declare `capabilities.controls` to keep receiving those values.
+- `runProviderSearch` / `providerCacheKey` / `stableStringify` extracted and exported (`provider-run`), shrinking the search orchestrator.

.changeset/architecture-review-mcp.md

@@ -0,0 +1,7 @@
+---
+"@refkit/mcp": minor
+---
+
+- `search_references` gains `rerank: true` (query-aware re-ranking via `lexicalReranker`, CJK-aware) and `cursor` (load-more pagination with cross-page dedup; the continuation token is returned as top-level `nextCursor` on every call, independent of `explain`).
+- BYOK provider packages are explicitly externalized from the tsup bundle so dynamic imports resolve from node_modules at runtime — `--omit=optional` actually omits them, and provider patch releases reach `@refkit/mcp` users without a republish.
+- BYOK provider packages moved from `dependencies` to `optionalDependencies` and are now loaded lazily, only when their key is present. Default installs (incl. `npx -y @refkit/mcp`) still get everything; installs with `--omit=optional` skip BYOK sources, and a key whose package is missing logs a stderr warning instead of crashing. `defaultProviders()` is now async.

.changeset/architecture-review-providers.md

@@ -0,0 +1,17 @@
+---
+"@refkit/provider-openverse": patch
+"@refkit/provider-pixabay": patch
+"@refkit/provider-internet-archive": patch
+"@refkit/provider-wikimedia-commons": patch
+"@refkit/provider-artic": patch
+"@refkit/provider-unsplash": patch
+"@refkit/provider-flickr": patch
+"@refkit/provider-smithsonian": patch
+"@refkit/provider-freesound": patch
+"@refkit/provider-jamendo": patch
+"@refkit/provider-europeana": patch
+"@refkit/provider-met": patch
+"@refkit/provider-polyhaven": patch
+---
+
+Declare and honor the `page` search control (`capabilities.controls: ['page']`), wiring `controls.page` to each source's native pagination — native `page` params where they exist, offset translation for offset-based APIs (Wikimedia `gsroffset`, Smithsonian/Europeana `start`, Jamendo/ambientCG `offset`), and a window over the full result list for Met/Poly Haven. Enables core's unified load-more cursor across these sources. (Brave, PoetryDB, and Rijksmuseum expose no usable offset pagination and keep `page` undeclared.)

.github/workflows/ci.yml

@@ -17,5 +17,9 @@ jobs:
           cache: pnpm
       - run: pnpm install --frozen-lockfile
       - run: pnpm typecheck
+      - run: pnpm lint
       - run: pnpm build
       - run: pnpm test:run
+      # source suites resolve workspace src/*.ts — only this exercises the bits we publish
+      - name: Artifact smoke (pack → clean install → boot MCP cli)
+        run: pnpm smoke:artifact

README.md

@@ -108,17 +108,17 @@ console.log(meta.providers)
 console.log(meta.warnings)
 ```
 
-`controls.page` is a **provider-local** cursor: each provider paginates its own result stream independently, and refkit does not track a unified offset across sources. Because pages are fused with Reciprocal Rank Fusion per request, page N+1 is not guaranteed to be disjoint from page N — results can overlap or shift relative to the previous page. For a "load more" UI, dedupe across pages by `canonicalUrl` rather than assuming stable, non-overlapping windows:
+### Pagination ("load more")
 
-```ts
-import { canonicalizeUrl } from '@refkit/core'
+Use the built-in cursor: every `searchWithMeta` result carries an opaque `meta.nextCursor`; pass it back as `cursor` and refkit advances the provider-local page **and dedupes against everything already returned** — no caller-side bookkeeping:
 
-const seen = new Set(prev.map((r) => canonicalizeUrl(r.canonicalUrl)))
-const nextPage = await refkit.search({ query, modalities: ['image'], controls: { page: 2 } })
-const newOnly = nextPage.filter((r) => !seen.has(canonicalizeUrl(r.canonicalUrl)))
+```ts
+const batch1 = await refkit.searchWithMeta({ query: 'forest path', modalities: ['image'] })
+const batch2 = await refkit.searchWithMeta({ query: 'forest path', modalities: ['image'], cursor: batch1.meta.nextCursor })
+// batch2 never repeats batch1; an empty batch (no meta.nextCursor) means exhausted.
 ```
 
-(core's own merge/dedup normalizes URLs the same way, so this recipe stays consistent with what refkit dedupes internally.)
+The cursor first **drains the overfetched pool** of the current provider page (each search fetches `limit × poolFactor` candidates but returns `limit`), and only then advances the provider-local page — so ranked results are never skipped. Under the hood `controls.page` is still provider-local (each source paginates its own stream; RRF-fused pages can overlap), which is exactly why the cursor tracks seen results for you. The seen set is capped (oldest evicted) so cursors stay small. Raw `controls.page` remains available if you want to manage pages yourself — then dedupe across pages by `canonicalizeUrl(r.canonicalUrl)`.
 
 ## Ranking & rerank
 
@@ -148,6 +148,10 @@ rerank: async ({ query, refs }) => myEmbeddingRerank(query, refs)
 
 Rerank is **opt-in** — omit it for the default RRF order. It runs post-merge, before the `gateFor` license filter and the limit.
 
+`lexicalReranker`'s term matching understands CJK text (character bigrams), so Chinese/Japanese/Korean queries score against titles instead of tokenizing to nothing. Note that most bundled sources index English metadata — for best recall, query in English (or have your agent translate) even though ranking handles CJK.
+
+When two sources disagree about the license of the **same canonical URL**, the merge resolves conservatively: the stricter license wins, and incomparable claims collapse to `unknown` (→ needs-review). Each conflict is reported in `meta.warnings` (and to an optional `merge.onRightsConflict` observer) — results never silently inherit the more permissive claim.
+
 URL dedupe is built in, and perceptual hashes are supported when providers or hosts supply them. For host-computed fingerprints or embeddings, add a duplicate hook without making core fetch or decode media:
 
 ```ts
@@ -170,10 +174,16 @@ createRefkit({ providers, resilience: { timeoutMs: 4000, retries: 2 } })
 createRefkit({ providers, resilience: false }) // raw fan-out, no timeout/retry
 ```
 
-Pass a `cache` to memoize **per-provider** results (keyed by provider + normalized query, TTL `cacheTtlMs`, default 5 min). Merging, reranking, and the license gate always run fresh; cache hits are flagged `cached: true` in `meta.providers`, and every provider status carries `latencyMs`:
+With many sources registered, bound the fan-out with `concurrency` — at most N provider searches run at once (a queued provider's timeout only starts when its slot starts):
+
+```ts
+createRefkit({ providers, concurrency: 6 }) // default: unlimited
+```
+
+Pass a `cache` to memoize **per-provider** results (short fixed-shape hashed keys, safe for strict KV backends; the cached value embeds the full normalized query and is verified on read, so a key collision degrades to a miss instead of serving another query's results; TTL `cacheTtlMs`, default 5 min). Merging, reranking, and the license gate always run fresh; cache hits are flagged `cached: true` in `meta.providers`, and every provider status carries `latencyMs`. Pass `cacheRaw: false` to strip each result's `raw` provider payload from cache entries (smaller entries; raw-reading `isDuplicate` hooks then won't see `raw` on hits):
 
 ```ts
-createRefkit({ providers, cache: myKvCache, cacheTtlMs: 60_000 })
+createRefkit({ providers, cache: myKvCache, cacheTtlMs: 60_000, cacheRaw: false })
 ```
 
 ## Providers
@@ -241,7 +251,7 @@ Agents can use refkit in two ways:
 npx -y @refkit/mcp
 ```
 
-It boots with the keyless sources (Met, Art Institute, Wikimedia, Openverse, Project Gutenberg, PoetryDB, Rijksmuseum, Poly Haven, ambientCG, Internet Archive) and auto-adds any BYOK source whose key is in the environment (`REFKIT_UNSPLASH_KEY`, `REFKIT_PEXELS_KEY`, `REFKIT_BRAVE_KEY`, … — legacy names like `UNSPLASH_KEY`, `PEXELS_KEY`, `BRAVE_TOKEN` still work as fallbacks). Pass `intent` to annotate each result with a use-verdict (may I use this, is attribution required); `gateFor` to return only allowed results. Beyond search, `evaluate_use` and `build_attribution` expose the same license-verdict and attribution logic as standalone stateless tools, for when an agent already has a license id and just needs a verdict or a credit line. Or wire your own providers/keys via `serveStdio(createRefkit({ … }))` — see [`@refkit/mcp`](https://www.npmjs.com/package/@refkit/mcp).
+It boots with the keyless sources (Met, Art Institute, Wikimedia, Openverse, Project Gutenberg, PoetryDB, Rijksmuseum, Poly Haven, ambientCG, Internet Archive) and auto-adds any BYOK source whose key is in the environment (`REFKIT_UNSPLASH_KEY`, `REFKIT_PEXELS_KEY`, `REFKIT_BRAVE_KEY`, … — legacy names like `UNSPLASH_KEY`, `PEXELS_KEY`, `BRAVE_TOKEN` still work as fallbacks). Pass `intent` to annotate each result with a use-verdict (may I use this, is attribution required); `gateFor` to return only allowed results; `rerank: true` for query-aware re-ranking (term coverage incl. CJK, resolution, source diversity); `cursor` (from the previous result's top-level `nextCursor` — always returned, no `explain` needed) to page through results without repeats. BYOK provider packages are `optionalDependencies` of `@refkit/mcp` — installed by default (zero-config `npx` keeps working), but an install with `--omit=optional` skips them, and a key whose package is missing just logs a stderr warning instead of crashing the server. Beyond search, `evaluate_use` and `build_attribution` expose the same license-verdict and attribution logic as standalone stateless tools, for when an agent already has a license id and just needs a verdict or a credit line. Or wire your own providers/keys via `serveStdio(createRefkit({ … }))` — see [`@refkit/mcp`](https://www.npmjs.com/package/@refkit/mcp).
 
 ## Not legal advice
 
@@ -252,6 +262,7 @@ It boots with the keyless sources (Met, Art Institute, Wikimedia, Openverse, Pro
 ```bash
 pnpm install
 pnpm typecheck   # all packages
+pnpm lint        # eslint (typescript-eslint, syntactic rules; tsc stays the type gate)
 pnpm test:run    # all packages
 pnpm build       # tsup → dist for every package
 ```

eslint.config.mjs

@@ -0,0 +1,31 @@
+// Deliberately lean: typescript-eslint's syntactic recommended set (no
+// type-checked rules — they'd re-do tsc's job slowly in CI). tsc --noEmit
+// remains the type gate; lint catches the bug-shaped patterns tsc allows.
+import tseslint from 'typescript-eslint'
+
+export default tseslint.config(
+  { ignores: ['**/dist/**', '**/node_modules/**', 'docs/**'] },
+  ...tseslint.configs.recommended,
+  {
+    rules: {
+      // `catch { /* fall through */ }` and `_`-prefixed placeholders are idioms here.
+      '@typescript-eslint/no-unused-vars': ['error', {
+        argsIgnorePattern: '^_',
+        varsIgnorePattern: '^_',
+        caughtErrors: 'none',
+      }],
+      // Providers cast `q.providerOptions as XSearchOptions` by design (typed whitelists).
+      '@typescript-eslint/consistent-type-assertions': 'off',
+      // zod schemas + Reference contracts legitimately use interface & type interchangeably.
+      '@typescript-eslint/no-empty-object-type': 'off',
+    },
+  },
+  {
+    files: ['**/__tests__/**', '**/*.test.ts'],
+    rules: {
+      // tests stub partial shapes and poke internals
+      '@typescript-eslint/no-explicit-any': 'off',
+      '@typescript-eslint/no-non-null-assertion': 'off',
+    },
+  },
+)

package.json

@@ -8,27 +8,31 @@
   "packageManager": "pnpm@10.32.1",
   "scripts": {
     "typecheck": "pnpm -r --parallel typecheck",
+    "lint": "eslint .",
     "build": "pnpm -r --if-present build",
     "test": "vitest",
     "test:run": "vitest run",
     "test:watch": "vitest watch",
     "test:live": "REFKIT_LIVE=1 vitest run",
+    "smoke:artifact": "node scripts/smoke-artifact.mjs",
     "changeset": "changeset",
     "version-packages": "changeset version",
     "release": "pnpm -r --if-present build && changeset publish"
   },
   "devDependencies": {
-    "@types/node": "^22.10.0",
-    "@resvg/resvg-js": "^2.6.2",
+    "@changesets/cli": "^2.27.0",
     "@refkit/core": "workspace:*",
+    "@refkit/provider-artic": "workspace:*",
+    "@refkit/provider-met": "workspace:*",
     "@refkit/provider-openverse": "workspace:*",
     "@refkit/provider-wikimedia-commons": "workspace:*",
-    "@refkit/provider-met": "workspace:*",
-    "@refkit/provider-artic": "workspace:*",
-    "tsx": "^4.22.4",
+    "@resvg/resvg-js": "^2.6.2",
+    "@types/node": "^22.10.0",
+    "eslint": "^10.7.0",
     "tsup": "^8.3.5",
+    "tsx": "^4.22.4",
     "typescript": "^5.7.2",
-    "vitest": "^3.2.6",
-    "@changesets/cli": "^2.27.0"
+    "typescript-eslint": "^8.64.0",
+    "vitest": "^3.2.6"
   }
 }