[e2e] Report partial results when the event-log race repro is cut short - #3148
Conversation
The repro harness wrote its result file once, after all planned attempts had landed, and derived its vitest timeout from `runTimeoutMs * ceil(attempts / concurrency)` — 141 min at the current defaults, above the job's own `timeout-minutes: 120`. The runner therefore always won the race: the job was cancelled, the `always()` reporting steps ran, found no file, and posted "No result file was produced by the latest repro job." Every run that had completed was discarded. Three changes: - The harness now owns a wall-clock launch budget (`EVENT_LOG_RACE_REPRO_BUDGET_MS`, default 75 min) checked before an attempt is claimed, never mid-attempt, and sliced per scenario so a truncated run still carries `hook-sleep` control data. The vitest timeout derives from the budget, so the harness finishes and reports before the job cap can fire. - Results are checkpointed to disk as attempts land (throttled to 10s), so a cancellation or timeout still reports the runs it paid for. - The result file carries `partial` / `budgetExhausted` / `plannedAttempts`, and the renderer says so in the headline and the history row rather than letting a short run read as a clean one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 0248496 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (1 failed)hono (1 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (1)8d5f54cTue, 28 Jul 2026 04:56:15 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Signed-off-by: Peter Wielander <mittgfu@gmail.com>
|
No backport to This is tooling work on the label-gated / manually dispatched event-log race repro harness, which is a diagnostic aid for an active investigation on To override, re-run the Backport to stable workflow manually via |
1 participant
The event-log race repro job posted "No result file was produced by the latest repro job." on both of its last two runs, discarding every attempt that had completed.
Cause
Two things compounded:
writeResultsran once, after all planned attempts had landed. A job that ends any other way produces no file at all. Thealways()reporting steps do run on cancellation — they just find nothing.runTimeoutMs * ceil(attempts / concurrency) + 60s= 141 min at the current defaults (1400 attempts, c40, 240s), above the job's owntimeout-minutes: 120. So the runner always won the race and vitest never reached its final write.Fix
EVENT_LOG_RACE_REPRO_BUDGET_MS(newbudget_msdispatch input, default 75 min) is checked before an attempt is claimed, never mid-attempt — an attempt that has started owns a real workflow run, and abandoning it would report a stuck run the harness itself caused. It is sliced per scenario in proportion to remaining planned attempts, so a truncated run still carrieshook-sleepcontrol data instead of dropping the last scenario entirely; a scenario finishing under its slice hands the surplus to the next. The vitest timeout now derives from the budget (budget + runTimeoutMs + 60s= 80 min), so the harness finishes and reports well inside the job cap.partial,budgetExhaustedandplannedAttempts; the renderer surfaces that in the headline and in the history row (… — partial (N of M planned)) so a short run cannot read as a clean one.--checkstill gates on regressions in a partial file.Tests
node --test .github/scripts/render-event-log-race-repro-results.test.js— 12/12, including four new cases: partial metadata passes throughbuildEntry, a complete file defaults to non-partial, a partial file renders as partial rather than missing, and--checkstill exits 1 on a regression in a partial file.Note
Branches that already have repro runs in flight (e.g. #3139) check out their own head SHA, so they need to merge
mainbefore this affects their runs.