feat(vitest-evals): support a direct meta.eval.input, mirroring meta.eval.output - #2318
Open
Aislinn Cronin (aislinnnnn) wants to merge 3 commits into
Open
Conversation
…eval.output The eval-level root span's `input` field could only be populated via harness.run.session.messages (the first message with role: "user"), via firstUserMessageContent(). That forces any non-conversational harness to fabricate a synthetic message just to surface its real input, even though meta.eval.output already supports a direct passthrough for the symmetric case, and logToolCallSpans in this same file already does `input: call.arguments` with no such indirection. Add meta.eval.input as a direct passthrough that takes precedence over session.messages when present, falling back to the existing firstUserMessageContent(run) behavior when it's not set — so existing consumers relying on session.messages are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
assigned
Abhijeet Prasad (AbhiPrasad) and unassigned
Abhijeet Prasad (AbhiPrasad)
The changeset described the internal shape (eval task meta, harness.run.session.messages) without ever naming the reporter it affects, unlike other changesets in this repo that name the concrete integration in the title. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
requested review from
Abhijeet Prasad (AbhiPrasad) and
Luca Forstner (lforst)
July 31, 2026 21:17
Member
|
thanks for the PR Aislinn Cronin (@aislinnnnn). We'll take a look at this on monday! |
The comment restated what the code and PR description already say. Test data used our internal domain fields (memberUuid, payPeriodEnd) instead of this file's existing refund/invoice fixtures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds
meta.eval.inputto thevitest-evalsreporter, as a direct passthrough for an eval row'sinputfield — the same waymeta.eval.outputalready works.Why
Today the eval-level root span's
inputcan only be populated viaharness.run.session.messages, specifically the first entry withrole: "user"(firstUserMessageContent()). That's a reasonable default for conversational harnesses, but it forces any harness whose input isn't a conversation (e.g. a structured payload, a single scalar) to fabricate a synthetic{ role: "user", content: ... }message just to get anything intoinputat all.This is inconsistent with the rest of this same file:
meta.eval.outputalready supports a direct passthrough (input.output !== undefined ? { output: input.output } : {}inreadEvalMeta) — there's no equivalent forinput.logToolCallSpansin this file setsevent: { input: call.arguments }directly, no message-shape required.Eval()framework (js/src/framework.ts) doesevent: { input: datum.input }— a direct, arbitrary passthrough, which is the norm for the rest of the SDK.Changes
EvalMetagains aninput?: unknownfield, alongside the existingoutput?: unknown.readEvalMetareadsinput.inputthe same way it already readsinput.output.logEvalTestcomputesconst input = meta.eval?.input ?? firstUserMessageContent(run);—meta.eval.inputtakes precedence when present, falling back to the existing session-derived behavior when it's not. Fully backward compatible: nothing changes for consumers that only setharness.run.session.messages.Testing
pnpm vitest run src/wrappers/vitest-evals/reporter.test.ts— all 6 tests pass (5 existing + 1 new).pnpm run fix:formatting— clean.minor, per the existing convention for new fields).🤖 Generated with Claude Code