feat(evals): wire opt-in OTEL transport (EVAL_TRACE_TRANSPORT=otel; native unchanged)

[miguelg719]

why

Turns the OTEL fan-out on behind an opt-in flag. The native (default) path stays byte-identical — no keys/flags means today's behavior exactly.

what changed

• framework/braintrust.ts: tracedSpan gains an otel branch — opens an OTEL active span, adapts span.log({output,scores,metrics,metadata}) → setAttribute/addEvent; native + NOOP_SPAN branches untouched, signature unchanged (verifier/runner call sites need zero edits)
• lib/AISdkClientWrapped.ts: EVAL_TRACE_TRANSPORT=otel → experimental_telemetry (no wrapAISDK); native path verbatim
• framework/runner.ts: provider init at run start; shutdownTracing() after the Braintrust flush
• cli.ts: best-effort flush on signal teardown

test plan

• Unit suite 362/362; braintrust-optional / braintrust-runner-nolog pass unchanged (native invariant)
• Manual: EVAL_TRACE_TRANSPORT=otel LANGSMITH_TRACING=true dual-backend run (env: browserbase) — spans land in both UIs; .trajectories/ identical to native

Stack 3/5 · base: #2352

🤖 Generated with Claude Code

---

Summary by cubic

Adds an opt-in OpenTelemetry transport for eval traces via EVAL_TRACE_TRANSPORT=otel. Native Braintrust tracing stays the default and unchanged.

• New Features

  • framework/braintrust.ts: tracedSpan adds an OTEL path that maps inputs/metadata/metrics to attributes and outputs/scores to events; records errors; native path untouched.
  • framework/runner.ts: builds the tracer provider at run start with a project-aware parent (project_name:*) when OTEL is enabled.
  • cli.ts: best-effort shutdownTracing() on signal teardown.
  • lib/AISdkClientWrapped.ts: when EVAL_TRACE_TRANSPORT=otel, call the SDK directly and pass experimental_telemetry for generateObject/generateText; native wrapping otherwise.

• Bug Fixes

  • framework/runner.ts: always calls shutdownTracing() in a finally block to avoid leaks.
  • Tests: added tracedspan-otel.test.ts to verify attribute/event mapping and error recording.

^{Written for commit 7e05a8a. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 7e05a8a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

All reported issues were addressed across 4 files

Confidence score: 5/5

• Safe to merge after the addressed issues were fixed.

Architecture diagram

sequenceDiagram
    participant CLI as CLI / Signal Handler
    participant Runner as runner.ts
    participant Braintrust as braintrust.ts (tracedSpan)
    participant OTEL as @opentelemetry/api (OtelSpan)
    participant AISDK as AISdkClientWrapped
    participant BraintrustAPI as Braintrust Backend
    participant OTELExporter as OTEL Exporter

    Note over CLI,OTELExporter: OTEL Transport (EVAL_TRACE_TRANSPORT=otel)

    Runner->>Runner: resolveTraceTransport() => "otel"
    Runner->>OTEL: buildTracerProvider()
    OTEL-->>Runner: provider ready

    Runner->>Braintrust: tracedSpan(fn, options)
    alt EVAL_TRACE_TRANSPORT equals otel
        Braintrust->>OTEL: getTracer().startActiveSpan(name)
        OTEL-->>Braintrust: active OtelSpan
        Braintrust->>OTEL: setAttribute(type, input.*)
        Braintrust->>fn: call with OTEL adapter
        fn->>AISDK: createChatCompletion()
        AISDK->>AISDK: loadAISDK() (no wrapAISDK)
        AISDK->>AISDK: pass experimental_telemetry block
        Note over AISDK: generates OTEL spans for generateObject/generateText
        AISDK-->>fn: response
        fn->>Braintrust: adapter.log({output, scores, metadata, ...})
        Braintrust->>OTEL: setAttribute(metadata.*, metrics.*)
        Braintrust->>OTEL: addEvent("scores", ...)
        Braintrust->>OTEL: addEvent("output", ...)
        alt error in fn
            Braintrust->>OTEL: setStatus(ERROR)
            Braintrust->>OTEL: recordException(error)
            Braintrust-->>Runner: throw error
        else success
            Braintrust->>OTEL: span.end()
            Braintrust-->>Runner: return result
        end
        Runner->>Braintrust: flush() (Braintrust log)
        Braintrust->>BraintrustAPI: send logs
        Runner->>OTEL: shutdownTracing()
        OTEL->>OTELExporter: export final spans
        OTEL-->>Runner: done
    else native (default)
        Braintrust->>Braintrust: hasBraintrustApiKey()
        alt has key
            Braintrust->>BraintrustAPI: create Braintrust span
            Braintrust-->>fn: Braintrust adapter
            fn->>Braintrust: adapter.log(...)
            Braintrust->>BraintrustAPI: log output/scores
        else no key
            Braintrust-->>fn: NOOP_SPAN
        end
        Braintrust-->>Runner: return result
    end

    Note over CLI,Runner: Signal teardown
    CLI->>CLI: handleSignal(SIGINT)
    CLI->>OTEL: shutdownTracing()
    OTEL->>OTELExporter: final flush
    CLI->>CLI: process.exit(code)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>