[STG-2515] feat(evals): vercel_ai_sdk bare-loop harness adapter

[shrey150]

What

Part 2 of 5 — stacked on #2337 (base: shrey/evals-bareloop-harnesses; GitHub shows only this PR's delta). When #2337 merges, retarget this PR's base to main and rebase.

Registers --harness vercel_ai_sdk: a bare loop via the ai package — generateText + stopWhen: stepCountIs(N) with a single gated browse tool. The AI SDK's own multi-step loop IS the harness; zero additional scaffolding. This mirrors the JS-default bare loop found in the wild (15.1M npm dl/wk) and the Modal sandbox template used in the 2026-07-09 smoke that surfaced 5 SKILL.md issues.

• Model resolution goes through stagehand's own getAISDKLanguageModel provider map, so -m provider/model means the same thing as on the stagehand harness.
• Step cap: shared 40 default, EVAL_VERCEL_AI_SDK_MAX_STEPS override.
• Tool calls recorded at execution time as NormalizedToolCalls → gradeExternalTrajectory unchanged.
• Testability: generateTextFn injection seam — unit tests drive the loop and invoke the real tool wiring against a temp browse binary.

External harnesses design + usage: packages/evals/README.md#external-harnesses (in #2337). Linear: STG-2515.

Comparability fix (2026-07-09 audit)

generateText returns normally whether stopWhen: stepCountIs(maxSteps) cut the loop off mid tool-call or the model actually finished — finishReason was declared on the result type but never read, so a cap-truncated run and a clean stop were indistinguishable in vercelAiSdkStatus/vercelAiSdkStopReason. Now reads finishReason === "tool-calls" at stepsUsed >= maxSteps as the step-cap signal and reports the same step cap reached (N) stopReason shape anthropic_sdk (#2339) already uses, so every bare-loop harness surfaces the cap identically. Two new tests cover both the capped case and a clean stop that happens to land exactly at the max step count (to prove the fix keys off finishReason, not just the step count).

E2E Test Matrix

Live smoke ran on the pre-split combined branch (content identical to this stack's tip — verified by diff); results transplanted, not rerun.

Command / flow	Observed output	Confidence / sufficiency
evals run b:webtailbench --harness vercel_ai_sdk -m anthropic/claude-haiku-4-5-20251001 -l 1 -t 1 -c 1 (local build, real ANTHROPIC + GEMINI keys, local Chrome via browse CLI)	Run completed (exit 0). 13 browse tool calls recorded (first action: open https://www.google.com); trajectory persisted; verifier graded: outcomeSuccess=false, processScore=0.79, 5 rubric criteria, no verifierError. Usage: 440k in / 1.3k out tokens	The bar for this smoke: harness completes, trajectory captured, verifier returns a graded outcome — all met. Task pass/fail is NOT the bar (united.com flights, hard target; haiku hit a browser hiccup mid-run and reported honestly)
pnpm exec vitest run (this branch)	Test Files 50 passed, Tests 371 passed — adds vercelAiSdkRunner.test.ts (system prompt = verbatim bare one-liner, tool wiring against a real temp binary, step/token metrics, error-to-TaskResult folding) + the registry test	Mocked-model loop coverage; live behavior proven by the smoke row above
tsc --noEmit + prettier + eslint	clean	Types/style only
pnpm exec vitest run (this branch, post-fix)	Test Files 50 passed, Tests 373 passed — adds the two step-cap tests (capped-with-tool-calls-finishReason vs. clean-stop-at-max-steps)	Proves the finishReason-based cap detection fires only on genuine truncation, not just on reaching the step count
pnpm -w exec tsc --noEmit + prettier + eslint (post-fix)	clean	Types/style only

The 440k cumulative input tokens in the smoke is what motivated the 20k-char tool-output clip that ships in the base PR (EVAL_BARE_LOOP_TOOL_OUTPUT_LIMIT) — the smoke ran pre-clip and still completed, but larger snapshot outputs would have overflowed context.

Tier 1 fixes (post-review audit, 2026-07-14)

An independent audit (on top of the 07-09 review round) found the step-cap status still reported "complete" and the token total still ignored the AI SDK's own totalUsage.totalTokens. Fixed here:

• Step-cap runs reported "complete", not "aborted". cappedOut was already detected correctly but never fed into status -- now cappedOut ? "aborted" : "complete".
• total_tokens recomputed input+output, ignoring the SDK's own total. VercelAiSdkGenerateTextResult.totalUsage now declares totalTokens, passed through via the base PR's new providerTotalTokens override instead of being silently recomputed.
• Updated the tool execute callback for BareLoopToolRecorder.execute()'s new { ok, output } return shape (base PR).

Verified: pnpm --filter @browserbasehq/stagehand-evals build && lint && test:unit -- Test Files 55 passed, Tests 420 passed (2 new tests for the totalTokens preference/fallback). Live: a real Browserbase local trial with EVAL_VERCEL_AI_SDK_MAX_STEPS=2 hit the step cap and the persisted trajectory (task_data.json) recorded "status": "aborted", not "complete".

No changeset: packages/evals is private.

🤖 Generated with Claude Code

---

Summary by cubic

Adds a vercel_ai_sdk bare-loop harness that runs ai.generateText with stopWhen: stepCountIs(N) and a single gated browse tool. Registers it as an executable bench harness; aligns model mapping with @browserbasehq/stagehand and reports step-cap truncation consistently with an aborted status and provider totalTokens passthrough (STG-2515).

• New Features

  • Registers --harness vercel_ai_sdk as executable (agent/suite).
  • Runs generateText with stopWhen: stepCountIs(N); default max steps 40 via EVAL_VERCEL_AI_SDK_MAX_STEPS.
  • Resolves models via @browserbasehq/stagehand getAISDKLanguageModel; requires provider/model.
  • Exposes a single browse tool; records tool calls; returns usage metrics (input/output, optional cached_input/reasoning, and provider totalTokens).
  • Supports an injectable generateTextFn for tests.

• Bug Fixes

  • Detects step-cap truncation using finishReason === "tool-calls" at the cap; sets status to aborted and stopReason step cap reached (N), matching other bare-loop harnesses.

^{Written for commit aaed51e. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: aaed51e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 5 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant CLI as eval CLI
    participant Registry as Harness Registry
    participant Runner as VercelAiSdkRunner
    participant AISDK as AI SDK (ai pkg)
    participant Stagehand as @stagehand/getAISDKLanguageModel
    participant Adapter as ExternalHarnessToolAdapter
    participant Browse as browse CLI (temp bin)
    participant Recorder as BareLoopToolRecorder
    participant Verifier as Verifier (gradeExternalTrajectory)

    Note over CLI,Verifier: NEW: vercel_ai_sdk bare-loop harness flow

    CLI->>CLI: Parse --harness vercel_ai_sdk
    CLI->>Registry: getBenchHarness("vercel_ai_sdk")
    Registry-->>CLI: vercelAiSdkHarness adapter

    CLI->>Adapter: prepareExternalHarnessAdapter(plan, model)
    Adapter->>Adapter: Build browse CLI path, env, system prompt
    Adapter-->>CLI: PreparedExternalHarnessAdapter

    CLI->>Runner: runVercelAiSdkAgent(input)

    Note over Runner: Resolve model provider

    Runner->>Runner: Check model includes "/"
    alt Missing provider prefix
        Runner->>Runner: Throw EvalsError("provider-prefixed model required")
    end
    Runner->>Stagehand: getAISDKLanguageModel(provider, modelName)
    Stagehand-->>Runner: LanguageModel instance

    Note over Runner: Read step cap config

    Runner->>Runner: readBareLoopMaxSteps(MAX_STEPS_ENV)
    Note right of Runner: Default 40, override with EVAL_VERCEL_AI_SDK_MAX_STEPS

    Note over Runner: Build tool recorder

    Runner->>Recorder: createBareLoopToolRecorder(adapter, logger)
    Recorder-->>Runner: recorder instance

    Note over Runner: Execute generateText loop

    Runner->>AISDK: import { generateText, tool, stepCountIs }
    Runner->>AISDK: generateText(options)
    Note right of Runner: model, system prompt, user prompt, stopWhen, tools

    AISDK->>AISDK: Run multi-step loop (stepCountIs(N) max)

    loop Per step (up to N)
        alt Model calls browse tool
            AISDK->>Runner: Execute tools.browse.execute({ args })
            Runner->>Recorder: recorder.execute(args)
            Recorder->>Browse: browse <args>
            Browse-->>Recorder: stdout result
            Recorder->>Recorder: Record normalized tool call
            Recorder-->>AISDK: tool output string
        else Model produces final text
            AISDK->>AISDK: finishReason = "stop"
        end
    end

    Note over AISDK: stopWhen cuts loop mid-tool-call

    AISDK-->>Runner: { text, steps, totalUsage, finishReason }

    Note over Runner: Detect step cap truncation

    alt stepsUsed >= maxSteps AND finishReason === "tool-calls"
        Runner->>Runner: cappedOut = true (step cap reached)
    else
        Runner->>Runner: Normal completion or clean stop
    end

    alt Loop threw error
        Runner->>Runner: capture error, log warn
        Runner->>Runner: status = "error", stopReason = error string
    end

    Note over Runner: Finalize result

    Runner->>Runner: finalizeBareLoopResult(...)
    Runner->>Recorder: recorder.calls (normalized tool calls)
    Runner->>Verifier: gradeExternalTrajectory(toolCalls)
    Verifier-->>Runner: verifier scores

    Runner-->>CLI: TaskResult with metrics
    Note right of Runner: vercelAiSdkStatus, vercelAiSdkStopReason, usage tokens

    CLI->>CLI: Write trajectory + scores

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>