fix(drift): stop gemini-interactions provider-mode false positive

[jpr5]

Root cause

The daily Drift Tests and Fix Drift jobs have been failing every run, firing a daily HTTP API drift detected ... providers changed response formats Slack alert — but there is no real drift.

The Gemini model-availability check (src/__tests__/drift/models.drift.ts) scrapes gemini-* tokens from source files (including README.md) with a greedy regex \b(gemini-(?:[\w.-]+))\b, then verifies each against Google's live model list. Commit 4753feb documented the aimock gemini-interactions provider mode in README.md:148 as a bare token. The scraper grabbed it as if it were a Gemini model id; the availability check failed (it is not a real Google model); and scripts/drift-report-collector.ts:~791 then fatally crashed (exit 1) because that AssertionError is neither a parseable drift block nor classifiable as infra.

The fix (two layers of defense)

1. Real fix — extend the Gemini stable filter to exclude aimock provider-mode names via an explicit AIMOCK_GEMINI_PROVIDER_MODES set, so any scraped provider-mode token is never drift-checked. Resilient to future doc mentions of the same mode.
2. Belt-and-suspenders — reword README.md so the provider mode is no longer written as a bare model-id token, so the greedy scraper does not pick it up in the first place.

Extracted scrapeModels / GEMINI_MODEL_PATTERN / sourceFiles / filterStableGeminiModels as exports and added src/__tests__/drift/models-scrape.drift.ts, a regression guard that exercises the real scrape + filter surface (no fakes) so a future doc mention re-introducing this false positive is caught here instead of in the live nightly job.

Red / Green (real surface, not fakes)

RED — production (already failing on main)

• Drift Tests (scheduled) run 29312286967 → failure
• Fix Drift run 29312327186 → failure

Verbatim from the failing drift job log:

Unparseable failure message (first 300 chars): AssertionError: Model gemini-interactions no longer available at Gemini: expected false to be true // Object.is equality
Fatal error: Error: 1 unparseable test failures with 0 drift entries — investigate

RED — local (pre-fix scrape + filter surface)

Replicating the exact pre-fix scrape + stable filter over the real README:

SCRAPED gemini tokens: ["gemini-2.0-flash","gemini-2.0-flash-exp","gemini-interactions"]
STABLE (post-filter):  ["gemini-2.0-flash","gemini-interactions"]

gemini-interactions survives the pre-fix filter — exactly what then gets checked against Google's model list, fails, and crashes the collector. A regression assertion (expect(preFixStable).not.toContain("gemini-interactions")) against this pre-fix state FAILS:

AssertionError: expected [ 'gemini-2.0-flash', …(1) ] to not include 'gemini-interactions'

GREEN — local (post-fix, same surface)

✓ src/__tests__/drift/models-scrape.drift.ts (5 tests | 3 skipped)
  Test Files  1 passed (1)
       Tests  2 passed | 3 skipped (5)

gemini-interactions is no longer treated as a model id; the false positive is gone.

GREEN — live workflow (authoritative; PR CI skips the live drift leg)

The drift job is guarded if: github.event_name != 'pull_request', so PR CI does not exercise this fix. Dispatched the live workflow_dispatch on this branch:

• Drift Tests run 29360718724 → conclusion: success — drift job passed in 39s, "Fail if critical drift detected" step skipped (no critical drift).

Quality gate

• pnpm run format:check — clean (only the git-ignored local .impeccable/hook.cache.json warns; not part of this change, not in CI)
• pnpm run lint — clean
• Tests-inclusive tsc --noEmit — zero errors in the changed files (models.drift.ts, models-scrape.drift.ts); pre-existing unrelated test-file type errors exist on main independently
• pnpm test — 147 files, 4364 tests, all pass
• pnpm test:drift — 15 passed / 10 skipped (no live keys locally; provider blocks skip); the new regression test passes
• pnpm build — clean

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@296

commit: 0874517