fix(core): order step-result deliveries against wait/hook deliveries by event-log position - #3139
Conversation
🦋 Changeset detectedLatest commit: f9c1006 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (5)285f434Tue, 28 Jul 2026 15:59:46 GMT · run logs
cc30335Tue, 28 Jul 2026 01:35:15 GMT · run logs
62f8c95Tue, 28 Jul 2026 00:00:19 GMT · run logs
e7ae7b3Mon, 27 Jul 2026 22:38:47 GMT · run logs
3fe83a1Mon, 27 Jul 2026 21:13:57 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
There was a problem hiding this comment.
Pull request overview
Fixes a replay-divergence class in @workflow/core where step result delivery order could flip relative to wait completions / hook payloads across cold vs warm replays (due to hydration/memoization timing), leading to deterministic ULID allocation mismatches and CORRUPTED_EVENT_LOG.
Changes:
- Extend the delivery-barrier registry to include
step_completed/step_failed, and defer step/wait/hook deliveries behind relevant earlier-in-log deliveries via a centralizedDEFER_BEHINDtable. - Add an
armedconcept for hook barriers so buffered (unclaimed) hook payloads don’t incorrectly block step deliveries; arm them onclaim(). - Add a regression test that replays the same committed event log twice against a shared
ReplayPayloadCacheto reproduce cold/warm ordering skew.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/workflow/sleep.ts | Update wait delivery deferral to include step deliveries via the new awaitEarlierDeliveries(ctx, idx, 'wait') behavior. |
| packages/core/src/workflow/hook.ts | Register hook barriers armed/unarmed based on whether a consumer is already awaiting; arm buffered payloads on claim(); defer behind earlier waits/steps. |
| packages/core/src/step.ts | Make step_completed and step_failed participate in delivery barriers, capturing hydrate outcome in-order but resolving/rejecting only after deferring behind earlier relevant deliveries. |
| packages/core/src/private.ts | Add step as a delivery kind, introduce DEFER_BEHIND, armed barriers, and resolvesOnItsOwn to avoid ordering steps behind unclaimed buffered hook payloads. |
| packages/core/src/step-delivery-ordering.test.ts | New cold/warm replay regression tests covering step-vs-wait and step-vs-hook ordering stability with a shared ReplayPayloadCache. |
| .changeset/step-delivery-ordering.md | Patch changeset documenting the behavioral fix for deterministic delivery ordering and divergence prevention. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
67e2f61 to
3fe83a1
Compare
|
Heads-up for anyone who read an earlier revision of this branch: the fix changed materially after I tried to verify a claim I was about to make in the description, and found it false. Summarising so the diff isn't confusing. The earlier revision was insufficient, and neither the repro tests nor the full suite could tell. I instrumented Two changes fix it properly:
And a mechanism correction to the shared test file's docstring, also from instrumentation: the
|
|
@TooTallNate handing this over to you — pausing the automated work here at a clean stopping point. State of the branch (head Your review items are intentionally unaddressed and yours to take:
Companion repro #3137 is final and green ( |
…by event-log position Two production runs on `@workflow/core@5.0.0-beta.36` burned all three divergence-recovery replays at the same event and terminated with CORRUPTED_EVENT_LOG: wrun_41KYJENABV0GSF5YTE9EETV5DD (step vs wait) wrun_41KYJEE01S0GPC9RWT5MEKVCX8 (step vs hook) Replay divergence: step event step_created for step_X belongs to "A", but the current step consumer is "B" `useStep` proxies draw deterministic ULIDs in invocation order, so the ULID -> stepName allocation is a function of the order in which promise resolutions are delivered to workflow code. The delivery-barrier registry pinned that order to event-log position for hook payloads and wait completions, but step results were delivered straight off the serial `promiseQueue` — and their latency varies between replays of the SAME invocation, because the first replay pays full hydration while later replays memo-hit primitive results in the shared `ReplayPayloadCache`. A step completion adjacent in the log to a `wait_completed` was therefore delivered wait-first on a cold replay and step-first on a warm one; whichever order the invocation that wrote the follow-up `step_created` events happened to see became law, and every replay computing the other order diverged permanently. Step results and step failures now register a 'step' delivery barrier at their event-log index and resolve from a detached continuation after every relevant earlier-in-log delivery, mirroring the hook payload path: hydration stays inside the serial queue slot (which also releases `pendingDeliveries`), while the barrier wait and the resolve run off the queue so a queue slot never blocks on a resolution the queue itself drives. Waits and hook payloads likewise defer behind earlier step results. Two details are what actually make the ordering hold, and both were found by testing rather than by reading the code: The deferral set is captured while CONSUMING the event, not at the start of the hydration slot. Captured at slot start it is not merely less deterministic, it is usually empty: an earlier delivery whose own slot runs first on the serial queue has typically already resolved and deregistered its barrier before the later slot begins, so the later delivery does not defer at all. Every event in one drain window is consumed before any slot runs, so consumption time sees all of them. A delivery that had to wait then yields a macrotask before resolving. An earlier delivery being "delivered" only means its `resolve()` ran; the branch it woke may need arbitrarily many further microtask hops before it reaches its next `useStep` call (a `for await` over a hook resumes the generator, settles the promise from `next()`, and only then runs the loop body). Ordering the `resolve()` calls alone therefore buys a fixed hop or two of margin and leaves a hop-count race that holds only for the shortest consumers; yielding a macrotask lets the earlier branch drain completely, whatever its shape. One asymmetry is load-bearing: a step result skips any earlier delivery that will not resolve on its own, i.e. one blocked directly or transitively on a buffered hook payload no consumer has claimed. Such a payload is delivered only when the workflow next reads the hook, and reaching that read commonly requires the step result itself, so gating the step on it stalls the run until the barrier's idle safety net fires — which then releases every delivery queued behind that payload at once and loses the very race the ordering exists to protect. Waits and hooks keep gating on unclaimed payloads, where waiting for the claim IS the guarantee. Tests come in two files. `step-delivery-ordering.test.ts` is byte-identical to the file in the repro-only companion PR #3137 apart from two `it.fails` markers there (which let a repro-only branch have green CI); `sed 's/it\.fails(/it(/g' | cmp` verifies it. Each of its five cases replays one committed log twice through a shared `ReplayPayloadCache`, and the two warm-replay cases fail on main with the production error text. `step-delivery-hop-count.test.ts` exists because those five cases cannot tell "delivered in log order" apart from "resolves a hop or two later than before". It replays logs a live run legitimately produced — the live invocation received the two events in separate deliveries, so the first branch finished long before the second event existed — while the replay receives both in one drain window, and pads the consumer with a varying number of extra awaits so hop count is the only variable. It covers step results against both wait completions and hook payloads, plus step FAILURES against wait completions, since a rejection decides whether a `catch` continuation runs and so which ULID the `useStep` there draws. All 18 cases fail on main; of the 12 that predate the macrotask, 9 still fail with the resolve-ordering-only version of this fix; all 18 pass here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Follow-up on the step-delivery barrier work, addressing three cases the registry did not yet cover. Each has a regression test in the new `delivery-barrier-coverage.test.ts` that reproduces the production `ReplayDivergenceError` when its fix is reverted. - Step results now defer behind earlier STEP results. The old exclusion assumed the serial `promiseQueue` fixes step-vs-step order, which stopped holding once a step began resolving from a detached continuation instead of its queue slot: two steps consumed in different drain windows can disagree on their deferral set, and the earlier one — parked on the macrotask yield — gets overtaken. - `sleep.ts` and `hook.ts` (waiting-consumer path) now capture their deferral at event-consumption time, as `step.ts` already does. Reading the registry after their queue work misses an earlier step or hook that delivered and retired its barrier in the meantime, skipping both the gate and the macrotask yield. The buffered hook payload path deliberately keeps evaluating at claim time; a consumption-time snapshot there stalls the e2e `hookWithSleepWorkflow`. - Abort deliveries participate in the registry. `_setAborted` fires the signal's listeners, which may invoke a step and draw a ULID, so an abort is as branch-deciding as any other delivery. Also memoizes `resolvesOnItsOwn`. The walk is exponential in the number of live hook/wait barriers, and the registry is not bounded — a fan-out of `Promise.race([hook, sleep])` branches accumulates one barrier per branch per kind (49 measured for 24 branches). At 40 barriers a single scan took 92s before, and is instant after.
|
Picking this up per @pranaygp's handoff above — rebased onto latest All three review items addressed, plus one more the abort work surfaced. Everything is covered by
One thing worth flagging as a design note rather than a fix: I initially applied the same consumption-time capture to the buffered hook payload's On your scrutiny list: item 1 (the Verification: unit Also dropped the SDK-drift reading of the Rollout caveat: runs are pinned to their originating immutable deployment on Vercel, so a run never replays across two |
|
Tarball soak test queued: vercel/o2flow#486 pins o2flow is currently throwing roughly one corrupted event log every 15–30 minutes, and its On that branch: typecheck clean, 2248 tests pass, build succeeds, and Correction on the plan: #486 has to merge to o2flow's Runs are pinned to the deployment that started them, so it's a clean cutover: in-flight runs stay on If the soak holds, this merges and ships as a normal Tracked in DPE-1721. |
Event Log Race Repro851 of 1400 latest repro runs hit event-log regressions. Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
Showing 20 of 851 non-completed runs. |
|
Backport PR opened against |
The bug
Two production runs on
@workflow/core@5.0.0-beta.36(o2flow) burned all three divergence-recovery replays at the sameeventIdand terminated withCORRUPTED_EVENT_LOG:wrun_41KYJENABV0GSF5YTE9EETV5DDReplay divergence: step event step_created for step_X belongs to "A", but the current step consumer is "B"wrun_41KYJEE01S0GPC9RWT5MEKVCX8Mechanism
useStepproxies draw deterministic ULIDs in invocation order (packages/core/src/step.ts:27), and replay pairs events to consumers bycorrelationId, erroring on astepNamemismatch (step.ts:88-100). So the ULID→stepName allocation is a pure function of the order in which promise resolutions are delivered to workflow code.The delivery-barrier registry (
ctx.pendingDeliveryBarriers,packages/core/src/private.ts) exists to pin that order to event-log position — its own docstring describes this exact failure class — but it only covered hook payloads and wait completions. Step results were outside it:step_completedresolved straight from its serialpromiseQueueslot, and that slot's latency varies wildly between replays of the same invocation:ReplayPayloadCacheand memo-hit small primitive results (replay-payload-cache.ts:125), resolving in one or two microtask hops.Meanwhile a
wait_completedresolves from a detached, fixed-hop chain (workflow/sleep.ts:100-116) that is about four hops off the queue tail. So a step completion adjacent in the log to await_completedis delivered wait-first on a cold replay and step-first on a warm one. Whichever order the invocation that wrote the follow-upstep_createdevents happened to observe became law; every replay computing the other order diverged permanently, and since the divergence is deterministic given a warm cache, all three recovery replays hit it at the same event.The fix
step_completedandstep_failednow participate in the barrier registry, mirroring the hook-payload path exactly:'step'barrier at the event's log index, so later wait/hook deliveries are handed over only after it;promiseQueueslot (async deserialization stays in log order) but only capture the outcome, releasingpendingDeliveriesin the slot'sfinally;markDelivered()+resolve/reject.Two details are what actually make the ordering hold. I found both by testing rather than by reading the code, and the first version of this PR had neither.
The deferral set is captured while consuming the event, not at the start of the hydration slot. Slot-start capture is not merely less deterministic — it is usually empty. An earlier delivery whose own slot runs first on the serial queue has typically already resolved and deregistered its barrier before the later slot begins, so the later delivery does not defer at all. Instrumenting the hook case showed exactly that:
awaitEarlier kind=step idx=4 defers=0, with the hook's barrier already gone. Every event in one drain window is consumed before any slot runs, so consumption time sees all of them (defers=1).A delivery that had to wait yields a macrotask before resolving. An earlier delivery being "delivered" only means its
resolve()ran. The branch it woke may need arbitrarily many further microtask hops before it reaches its nextuseStepcall and draws a ULID — afor awaitover a hook resumes the generator, settles the promise fromnext(), and only then runs the loop body; workflow code may await anything in between. Ordering theresolve()calls alone therefore buys a fixed hop or two of margin and leaves a hop-count race that holds only for the shortest consumers. Yielding a macrotask lets the earlier branch's whole microtask chain drain first, whatever its length. Only deliveries that actually deferred pay it, so the common single-delivery drain is unaffected.wait_completedandhook_receiveddeliveries symmetrically defer behind earlier step results.step_failedis included because a rejection is just as branch-deciding as a success — it decides whether acatchcontinuation runs, and therefore which ULIDs follow-upuseStepcalls draw. It also now holdspendingDeliveriesacross hydration, which it previously did not.awaitEarlierDeliveries(ctx, index, kind)now takes the delivering kind instead of an explicit kind list, with a singleDEFER_BEHINDtable as the source of truth:hookwait,stepwaithook,stepstepwait,hookpromiseQueueDeadlock / cycle analysis
sleep.tsalready documents).pendingDeliveriesis released inside the slot, before the detached defer — otherwisescheduleWhenIdlecould never reach idle, and the idle safety net that retires abandoned barriers (which this deferral may be waiting on) would never fire. This is the same shape as the hook path and the reasonabort-controller.tsbumps the counter.awaitEarlierDeliveriesresolves immediately with no macrotask, so the step'sresolveis enqueued at the end of its own slot, before the next queue slot runs. Ordering against every other queue-slot-driven delivery (includingabort-controller.ts's_setAborted, which is not a barrier participant) is therefore unchanged from before the fix, and no macrotask is paid.setTimeout(0)awaited in the detached continuation, afterpendingDeliverieshas already been released in the slot, so it neither blocks the serial queue nor preventsscheduleWhenIdlefrom reaching idle.scheduleWhenIdlealso runs onsetTimeout(0)andpendingDeliveriesis already 0 by then. The deferred step's macrotask is registered the moment the earlier barrier is marked delivered — i.e. before the woken branch's continuation runs at all — whereas the suspension macrotask is only registered once some branch reaches auseSteppast the end of the log, which happens within that continuation's microtask chain. Registration order therefore puts the step's resolve first, and its own continuation (a microtask) draws its ULID before the suspension macrotask fires. The 18 hop-count cases exercise exactly this: each ends in a suspension and asserts that both follow-up steps are pending, which only holds if the deferred delivery landed before the suspension took effect. This does assume workflow continuations are microtask chains, which holds because the sandbox has no real timers —sleep()is a wait event, not asetTimeout.registerDeliveryBarrier→scheduleWhenIdle(finish)is unchanged.One deviation from the obvious design, and it is load-bearing
Gating a step result behind every earlier hook barrier — the straightforward reading — is wrong, and the existing suite catches it. A buffered hook payload (no consumer awaiting when
hook_receivedis consumed) is delivered byclaim(), i.e. whenever the workflow next reads the hook — which is very commonly downstream of the step result itself (await setupStep()and only theniterator.next()). Gating the step on it stalls the run until the barrier's idle safety net fires, which then releases the step and every delivery queued behind that payload at once; the workflow installs itsPromise.raceagainst an already-resolved sleep and loses the very race the ordering exists to protect.Implemented naively, that regressed 4 existing tests:
So barrier entries carry an
armedflag:waitandstepare always armed, a hook payload with a waiting consumer is armed at registration, and a buffered payload is registered unarmed and armed byclaim(). A step result skips any earlier delivery that will not resolve on its own — unarmed, or transitively blocked on something unarmed (resolvesOnItsOwn). Waits and hooks deliberately keep gating on unclaimed payloads: for them, waiting for the claim is the ordering guarantee (await_completedmust not preempt a payload the log ordered first). Those 4 tests are now a real regression guard for this asymmetry.Test evidence
packages/core/src/step-delivery-ordering.test.tsis byte-identical to the file in the repro-only companion PR #3137 apart from twoit.failsmarkers there, which let a repro-only branch have green CI (sed 's/it\.fails(/it(/g' | cmpverifies it). So the tests demonstrably were not bent to fit the fix. Each case replays one committed event log twice through a sharedReplayPayloadCache(production shape: one cache per queue delivery, many replays), withhydrateStepReturnValuestubbed to cost 10ms so only the cold replay pays it.Five cases, two of which fail on
main:mainwrun_41KYJENABV0GSF5YTE9EETV5DDfor awaithook vs step, cold replayfor awaithook vs step, warm replaywrun_41KYJEE01S0GPC9RWT5MEKVCX8await hookvs step, both replaysThe hook flavour must use
for await (… of hook)to reproduce, but not because the payload is buffered — I instrumentedworkflow/hook.tsto check, and all three hook cases consumehook_receivedwith an awaiter already registered (promises.length === 1);claim()never runs in any of them. Enteringfor awaitcallsnext(), which reachesyield await thisand subscribes well before the events consumer drains the log onprocess.nextTick.What actually differs is how many microtask hops separate the payload resolving from the branch calling
afterHook()and drawing its next ULID.for awaitresumes the async generator atyield await this, settles the promise returned bynext(), and only then runs the loop body; a memo-warmstep_completed— which on an unfixed runtime resolves inside its own queue slot with no detached chain at all — gets there first. A directly-awaited hook resumes its continuation in the first microtask and stays ahead even on a warm cache, which is why it makes a good control: same delivery path, same event log, opposite outcome onmain, and this PR changes the code it runs through (the barrier is now registeredarmedthere), so it must stay green.So the hook flavour is the same hydration-latency race as the wait flavour; the consumer shape only sets how much of a head start the step result needs in order to win. (Buffered payloads are central to the
armeddesign below, but they are exercised byhook-sleep-interaction.test.ts's queued-payload cases — verified by the same instrumentation:promises.length === 0, thenclaim()→arm()— not by these new tests.)Without the fix (fix sources reverted to the branch's base commit, converged test file kept):
This matches the repro PR's independently observed baseline on
mainexactly — same two cases, same ULID, same error text — and it reproduces the same way in CI, on both Linux and Windows: in #3137's run the two warm-replay cases satisfy theirit.failsmarkers while the other three pass, i.e. the divergence is not an artifact of one machine's timing.The second test file, and why these five cases are not sufficient
All five cases above are consumed by branches that reach their next
useStepcall in the fewest possible microtask hops. That makes them unable to distinguish two very different guarantees: (a) step results are delivered in event-log order relative to wait and hook deliveries, or (b) a step result merely resolves a hop or two later than it used to, which beats the shortest consumers and nothing else.The distinction is not academic, and I only caught it by probing: the first version of this PR provided (b) and passed all five cases anyway.
packages/core/src/step-delivery-hop-count.test.tscloses that hole. Each case replays a log a live run legitimately produced — the live invocation received the two events in separate deliveries, so the first branch ran to completion long before the second event existed — while the replay receives both in one drain window and must still allocate the ULIDs in the recorded order. The consumer is padded with 0, 1, 2, 4, 8, or 16 extraawaits so hop count is the only variable, across three groups: step result vs wait completion, step result vs hook payload, and step failure vs wait completion (a rejection decides whether acatchcontinuation runs, and so which ULID theuseStepthere draws —step_failedgoes through the same registration and capture, so it needs the same guard).main(no fix)The middle row is the point: without the event-consumption-time capture and the macrotask yield, a consumer with a single extra
awaitafter being resumed still diverges. These cases are the regression guard for the guarantee actually being (a).With the fix:
5 passed (5)and18 passed (18). The 15 ordering-sensitive files (both new files plushook-sleep-interaction,async-deserialization-ordering,abort-replay-ordering,abort-consistency,step-hydration-memoization,workflow,runtime,events-consumer,step,workflow/sleep,workflow/hook,define-hook,runtime/resume-hook) were run 8× consecutively, stable at321 passed | 3 expected failevery time.Full core suite (74 files):
1565 passed | 3 expected fail— the 3 expected-fails are pre-existingit.failscases. Under a colour TTY,log-format,logger, andcontext-errorsadditionally report ANSI inline-snapshot mismatches; those fail identically on the unmodified tree and pass underFORCE_COLOR=0, i.e. a local artifact, not a regression.pnpm --filter @workflow/core typecheckis clean, andbiome checkreports only the twonoExcessiveCognitiveComplexitywarnings thatstep.tsalready carries onmain(63 and 20 there, 63 and 21 here).Full local e2e against a
nextjs-turbopackdev server, re-run after the macrotask change since it shifts real timing:135 passed (135)— the whole suite, including everyAbortControllercase,cancelRun,hookWithSleepWorkflow,hookWithSleepFinalStepWorkflow,sleepWithSequentialStepsWorkflow, andsetAttributes.CI history. An earlier revision of this branch (
3fe83a19, same barrier design, resolve-ordering only) went green across the whole matrix: all 13E2E Vercel Prod Testsapps,E2E Local Dev,E2E Local PostgresandE2E Local Prod(three world backends, ~10 frameworks each),Unit Testson Linux and Windows,Lint,Docs Checks,Tarballs Checks,Performance BenchmarksandDCO— tallySUCCESS: 80, SKIPPED: 6, FAILURE: 3. Revisiond4f86aa4(converged test file) likewise passedUnit Testson both Linux and Windows, with the Linux core-suite line matching my local numbers exactly.The 3 failures in that tally were
E2E Local Prod Tests (nextjs-webpack - stable),(nuxt - stable), and theE2E Required Checkgate aggregating them: all the same pre-existing flake,webhookWorkflowhitting its 120s timeout at1 failed | 153 passed (154).mainfails identically at commit4ada27d3(same test, same timeout, same counts), so it is unrelated to this change. For the record, the first run on this branch also tripped the documented rotatingE2E Vercel Prodflake on 7 apps;mainat this branch's exact base commit4ba223a0tripped it on 9, and the re-run came back fully green.CI is re-running on the current revision, which adds the event-consumption-time capture, the macrotask yield, and the hop-count test file.
The companion repro-only PR is #3137 (branch
pgp/repro-step-wait-divergence): it adds this same test file without the fix, and its CI confirms on Linux and Windows that the two warm-replay cases diverge onmainwhile the other three hold. Merging that first, then this, gives a red-then-green history for the bug; whoever lands them should flip those two cases back fromit.failsto plain assertions as part of merging this PR.Follow-up: remaining barrier gaps (commit 2)
Review of the first commit turned up three more places the registry did not
reach. Each is fixed here, with a regression test in the new
packages/core/src/delivery-barrier-coverage.test.tsthat reproduces theproduction
ReplayDivergenceErrorwhen — and only when — its own fix isreverted. Verified one at a time.
1. A step must defer behind earlier STEP results
DEFER_BEHIND.stepexcluded'step'on the grounds that the serialpromiseQueuealready orders step results. That stopped being true in thisPR: a step captures its outcome in its queue slot but resolves from a detached
continuation once its barrier clears. Two steps agree on that ordering only
while they defer behind the same set — true within one drain window, false
across windows. A step consumed later can miss a wait/hook barrier an earlier
step is still parked on, because it retired in between; the earlier step is
then waiting out the macrotask yield while the later one resolves on
microtasks.
The test log is one a live run legitimately produces:
stepA's worker writesstep_completedjust before the invocation resuming from a sleep writesstep_created stepB, so that invocation's own replay never deliveredstepA.On replay
stepB's consumer does not exist until the sleep lands, so the twocompletions fall in separate drain windows.
Cost of the extra edge, measured on a parallel step fan-out (
Promise.allofN steps, all completing in one drain): +1.3 ms, flat, not O(N) — the idle
safety net retires the chain in one timer tick rather than serializing N of
them. N=10 3.1→4.4 ms, N=50 3.1→5.2 ms, N=100 6.6→7.8 ms.
2.
sleep.tsandhook.tsmust capture their deferral at consumption timeThe "Coverage" argument the first commit documents for
step.tsappliessymmetrically, and was not applied.
sleep.tsread the registry after thequeue tail and
hook.tsat the end of its hydration slot — by which point anearlier step or hook has usually delivered and deregistered its barrier. The
later delivery then skips both the gate and
awaitEarlierDeliveries'macrotask yield, and overtakes the branch that earlier delivery just woke.
This is the exact mirror of
step-delivery-hop-count.test.tsand fails thesame way: a wait completion diverges at 8 extra consumer hops, a hook payload
at 20. Both are covered at 0/1/3/8/20/50 hops in the new file.
The buffered hook payload path deliberately keeps evaluating at claim
time, and now says so. A buffered payload's delivery genuinely happens when
the workflow reads the hook, potentially many deliveries later; a
consumption-time snapshot makes the claim wait on barriers from a moment it
never participated in, which stalls the second payload in the e2e
hookWithSleepWorkflowlong enough for the run to suspend before deliveringit. Caught by the full local e2e run, not by unit tests.
3. Abort deliveries participate in the registry
abort-controller.tsresolved_setAbortedstraight off itspromiseQueueslot and registered no barrier. That was sufficient only while every other
delivery also resolved from its slot.
_setAbortedfires the signal'slisteners, and a listener may invoke a step and draw a ULID, so an abort is as
branch-deciding as any other delivery — a log-earlier step sitting behind a
barrier is overtaken by any abort whose slot runs in the meantime. It now
registers as a
'hook'(which is exactly what the event is), always armed,and resolves detached behind its deferral.
Also:
resolvesOnItsOwnwas exponentialEach armed entry re-walked every earlier entry of the opposite kind —
T(n) = Σ T(j)— with no memo, on the synchronous delivery path. The registryis not bounded:
EventsConsumerdrains consecutively consumable eventssynchronously while barriers only retire on microtask-driven deliveries, so a
fan-out of
Promise.race([hook, sleep(watchdog)])branches accumulates onebarrier per branch per kind (49 live barriers measured for 24 branches).
A single scan over 40 alternating armed hook/wait barriers took 92 seconds
of blocked event loop before the fix, and is instant after. Guarded by a test
in the new file that times the synchronous scan.
Verification (commit 2)
pnpm vitest run --root packages/core src/→ 75 files, 1614 passed | 3 expected fail. Typecheck clean. Biome: same 8 pre-existing complexity/banned-type warnings as the unmodified tree, none new.277 passed | 3 expected failevery time.nextjs-turbopackdev server: 135 passed (135).e2e/local-build,e2e/build-errors,e2e/route-bundle-isolationpass individually. Run as one vitest invocation,route-bundle-isolationfails — identically on the unmodified tree, so it is cross-suite build interference, not a regression.Rollout caveat
In-flight runs whose logs were allocated under the old flipped (step-first) ordering will now deterministically diverge on replay. Those runs were already unrecoverable time bombs under the old runtime — any warm replay computed the other order, which is exactly how these two runs died — so this bounds the blast radius to windows that were already affected rather than creating a new one.
Not a concern for SDK version drift specifically: on Vercel a run is pinned to its originating immutable deployment, so a single run never replays across two different
@workflow/coreversions. The caveat above is only about logs already written under the old ordering, replayed by the same deployment.What reviewers should scrutinise
armedasymmetry (private.ts,resolvesOnItsOwn+awaitEarlierDeliveries): is "a step never queues behind an unclaimed buffered payload" the right line, or should the wait/hook side eventually adopt it too? Today they must not, or the original hook-vs-sleep fix regresses.arm()timing — the thinnest part of the argument. A buffered payload is armed atclaim(), and a step now reads armed-ness while consuming its event, which is exactly log position and cannot depend on hydration duration. The residual sharp edge is a shape where the claim is armed by a continuation of an earlier delivery, i.e. where the arming itself is latency-coupled one level up, so a step consuming its event might see the entry unarmed on one replay and armed on another. I could not construct a case that diverges, and none of the existing tests exhibit it — the new tests never buffer, andhook-sleep-interaction's queued-payload cases claim from a continuation downstream of a step but with only one step delivery in flight. So treat this as unproven rather than verified; it is where I would look first if a similar divergence resurfaces.setTimeout(0)per step delivery (cheap; the poll exits the first timependingDeliverieshits 0), and the new yield adds another only when a delivery actually had to defer — i.e. only when two or more branch-deciding events land in the same drain window, which is rare and is precisely the case that was broken. A sequential single-step workflow pays neither. Still worth a look given the active inline-step perf work, and worth challenging if anyone sees a cheaper way to let a resumed branch quiesce than a macrotask.step_failednow holdspendingDeliveriesacross hydration (it did not before), so suspensions wait for a rejection to land. Intentional and consistent withstep_completed, but it is a behaviour change beyond pure ordering.🤖 Generated with Claude Code