refactor(evals): extract EvalRunner interface (Braintrust Eval() default; no behavior change)

[miguelg719]

why

Isolates the Braintrust Eval() coupling behind a thin seam so a future backend swap is mechanical instead of a rewrite. Pure structural refactor.

what changed

• framework/evalRunner.ts (new): EvalRunner interface + BraintrustEvalRunner wrapping the existing Eval() call verbatim (same progress/reporter/noSendLogs/flush semantics)
• framework/runner.ts: runEvals builds a config and delegates; everything else (trajectories, summary mapping, otel init/shutdown, offline mode, abort handling) unchanged

test plan

• Unit suite 362/362; concurrency/trialCount/results/summary shapes verified unchanged in review
• Offline (noSendLogs) behavior identical

Stack 4/5 · base: #2353

🤖 Generated with Claude Code

---

Summary by cubic

Extracted an EvalRunner interface and a default BraintrustEvalRunner to decouple eval execution from Braintrust. No behavior change; runEvals now builds a config and delegates via the interface.

• Refactors
  • Added framework/evalRunner.ts with EvalRunner, EvalRunnerConfig, EvalRunnerResult, and BraintrustEvalRunner that wrap Braintrust Eval()/flush, use internal silent progress/reporter, and respect noSendLogs.
  • Updated framework/runner.ts to build the runner config, pass sendLogs via hasBraintrustApiKey(), and delegate execution; abort logic, offline mode, concurrency/trials, and summary mapping are unchanged.

^{Written for commit 24f88b9. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 24f88b9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

No issues found across 2 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Runner as runEvals()
    participant EvalRunner as EvalRunner Interface
    participant Braintrust as BraintrustEvalRunner
    participant BraintrustLib as @braintrust/core (Eval/flush)
    participant Task as task callback
    participant Progress as onProgress callback

    Note over Runner,Progress: Eval execution flow (no behavior change)

    Runner->>EvalRunner: run(config)
    activate EvalRunner
    EvalRunner->>Braintrust: Eval(projectName, {experimentName, metadata, data, task, scores, maxConcurrency, trialCount}, {noSendLogs})

    Braintrust->>Task: invoke for each testcase
    activate Task
    alt cooperative abort active
        Task->>Progress: {type: "failed", error: "aborted"}
        Task-->>Braintrust: {_success: false, error: "aborted by user"}
    else normal execution
        Task->>Progress: {type: "started"}
        Task->>Task: executeTask(input, resolvedTask, options)
        alt success
            Task->>Progress: {type: "passed"}
        else failure
            Task->>Progress: {type: "failed", error}
        end
        Task-->>Braintrust: TaskResult
    end
    deactivate Task

    Braintrust-->>EvalRunner: evalResult
    alt sendLogs == true
        EvalRunner->>Braintrust: flush()
        Braintrust-->>EvalRunner: done
    end
    EvalRunner-->>Runner: EvalRunnerResult
    deactivate EvalRunner

    Note over Runner,Progress: Post-eval (unchanged)
    Runner->>Runner: shutdownTracing() if otel

Loading

_{Re-trigger cubic}