[utils] Fix vercel world not being selected when running build on external CI - #3144
Conversation
World selection moved from runtime to build time, but the detection rule still keys only on VERCEL_DEPLOYMENT_ID. That variable does not exist while `vercel build` runs — no deployment has been created yet — so prebuilt pipelines (`vercel build` + `vercel deploy --prebuilt`) compile the local filesystem world into the deployment, and every run fails writing to a read-only filesystem. Fall back to VERCEL=1, which is present whenever system environment variables are exposed, and exclude development so that `VERCEL=1` arriving in .env.local via `vercel env pull` does not flip a dev server onto the Vercel world. Also warn from the local world when it initializes inside a Vercel deployment with a data directory outside /tmp, so a deployment built against the wrong world says so instead of failing on filesystem writes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 6b32304 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📜 Previous results (2)9d62ee2Tue, 28 Jul 2026 00:04:59 GMT · run logs
fa7d14eMon, 27 Jul 2026 23:01:15 GMT · run logs
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests📦 Local Production (1 failed)nextjs-webpack-stable (1 failed):
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
❌ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
VaguelySerious
changed the title
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Reviewed with the Slack thread context — fixing the build-time inference signal (rather than reverting #2752) makes sense, and the verification here is great: real vercel build against a linked project, bundle greps, and a before/after repro of the customer failure.
One substantive concern, inline on world-target.ts: the "ambiguity resolves toward Vercel because that failure is loud" premise doesn't hold when pulled production env includes VERCEL_OIDC_TOKEN — that direction can silently point a local next start at production. The rest are minor.
Two things that shouldn't get lost when the #3142/#3143 stack closes:
- #3143's runtime hardening (actionable errors when a custom world package can't load, bundler-safety tests) is valuable independent of the build-vs-runtime decision.
- The
agents/feedbackfailure (Workflow target world was not statically injectedfrom a server-action bundle) is an alias-coverage bug in the same mechanism — this PR fixes the inference signal but not that, so it needs its own fix.
| * framework dev server). Ambiguity resolves toward Vercel because that failure | ||
| * mode is loud — the Vercel world reports missing credentials — whereas | ||
| * wrongly choosing the local world silently writes workflow state to a | ||
| * read-only filesystem. |
There was a problem hiding this comment.
The "fails loudly" premise doesn't hold in the most realistic instance of the ambiguity: vercel env pull --environment production writes VERCEL_OIDC_TOKEN into .env.local, and world-vercel authenticates via getVercelOidcToken() (packages/world-vercel/src/utils.ts:307-313), which reads it. So next build && next start on a dev machine with pulled prod env selects the Vercel world, authenticates successfully, and starts/executes workflows against production state (with step code running locally) for the OIDC token's validity window — no credential error at all. Pre-#2752 runtime selection resolved this same setup to the local world, so this is a new exposure, not a pre-existing one.
Two cheap mitigations (either or both):
- A symmetric runtime warning in world-vercel — the mirror of this PR's world-local warning: warn once when the Vercel world is created in a process with no
VERCEL_DEPLOYMENT_ID. Then both wrong-world directions are observable. - A build-time log line when the Vercel world is selected via the
VERCEL=1fallback rather than a deployment ID, namingWORKFLOW_TARGET_WORLD=localas the opt-out — makes the new inference visible in CI logs instead of only in bundle greps.
There was a problem hiding this comment.
(AI) Checked this empirically, and the exposure doesn't exist — but for a reason that made the message the real defect, so this pushed the PR in the direction you asked for anyway.
start() awaits world.getDeploymentId() before it writes anything (packages/core/src/runtime/start.ts:278, ahead of run creation and unconditional even when opts.deploymentId is given), and world-vercel's getDeploymentId() reads process.env.VERCEL_DEPLOYMENT_ID and throws when it is unset — before any credential is used. So next build && next start with pulled prod env cannot reach production state regardless of how valid VERCEL_OIDC_TOKEN is. I verified it on the workbench app: build confirmed as world-vercel (0 world-local markers / 37 world-vercel markers in the server bundles), then POST /api/start with a deliberately valid-shaped env returned 500 {"details":"VERCEL_DEPLOYMENT_ID environment variable is not set"} and no run was created.
What was wrong is that message: it names an env var and nothing else — no mention that the world was chosen at build time, or that WORKFLOW_TARGET_WORLD=local + rebuild is the fix. That's the loud-failure claim in this comment block failing in practice. Fixed in 9d62ee2: packages/world-vercel/src/deployment-id.ts holds one message used by getDeploymentId(), queue(), and resolveLatestDeploymentId(), and the doc comment here now points at that throw instead of at credentials. Docs got the mirror-image paragraph too.
On the two mitigations:
- Declined as written. A warning at world-creation time fires during builds, not just at runtime:
createWorld()runs at build time viagetWorldHandlers(), where a prebuilt build legitimately has no deployment ID — so everyvercel buildwould print it. The throw ingetDeploymentId()covers the same wrong-world direction at the only moment it is unambiguously wrong. - Reasonable, not in this PR.
ensureWorkflowTargetWorldEnv()is called several times per build (and with synthetic env objects from tests), so a log line needs a once-guard with a test seam. The inference is now documented inconfiguration/worlds.mdxand both wrong-world directions fail with an actionable message; happy to add the log line as a follow-up if you still want it visible in CI logs.
| normalizeWorkflowTargetWorldImport(resolveWorkflowTargetWorld(env)) ?? | ||
| '@workflow/world-local' |
There was a problem hiding this comment.
nit: the ?? '@workflow/world-local' arm is unreachable — resolveWorkflowTargetWorld always returns a non-empty string, and normalizeWorkflowTargetWorldImport only returns undefined for falsy input. It exists solely to satisfy the string | undefined return type; an explicit non-optional path would be clearer than a dead fallback.
There was a problem hiding this comment.
(AI) Fixed in 9d62ee2 — the fallback is gone, replaced by a cast with a comment naming why the type is wider than the values:
// `resolveWorkflowTargetWorld` always returns a non-empty specifier, which is
// the only input `normalizeWorkflowTargetWorldImport` maps to `undefined`.
return normalizeWorkflowTargetWorldImport(
resolveWorkflowTargetWorld(env)
) as string;| if (process.env.VERCEL_DEPLOYMENT_ID === undefined) { | ||
| // Keep the generated manifest out of the developer's repository. A | ||
| // Vercel build has no repository to dirty. | ||
| if (!isVercelDeploymentEnv()) { |
There was a problem hiding this comment.
This site is different from the BOA one: the public manifest is written into the app's own public/.well-known/workflow/v1, and a prebuilt build (vercel build locally or in CI — the exact case this PR fixes) does run in a checked-out repository. With WORKFLOW_PUBLIC_MANIFEST=1, the generated manifest can now dirty the repo where it previously got a .gitignore. The comment's premise ("a Vercel build has no repository to dirty") is wrong precisely for prebuilt builds.
Suggest keeping this write unconditional — it's harmless in Vercel's build container and protective everywhere else.
There was a problem hiding this comment.
(AI) You're right, and it's worse than dirtying the repo: with WORKFLOW_PUBLIC_MANIFEST=1 a prebuilt build would have stopped writing the .gitignore into a real checkout. Reverted to the VERCEL_DEPLOYMENT_ID guard in 9d62ee2 (unchanged from main), along with the wrong comment.
| if (process.env.VERCEL_DEPLOYMENT_ID === undefined) { | ||
| // Keep the generated manifest out of the developer's repository. A | ||
| // Vercel build has no repository to dirty. | ||
| if (!isVercelDeploymentEnv()) { |
There was a problem hiding this comment.
The change here is fine, but for a different reason than the comment gives: prebuilt builds do run in a repository — it's just that .vercel/output is generated output (and the CLI gitignores .vercel on link), so this .gitignore never protected anything. Skipping it means prebuilt static output stops shipping a stray .gitignore as a deployable asset, matching Vercel-container builds. Suggest the comment say that instead of "no repository to dirty".
There was a problem hiding this comment.
(AI) Reverted this site to main as well in 9d62ee2, rather than keeping the change with a corrected comment. It isn't needed for the bug this PR fixes, and your reading is the accurate one: the .gitignore never protected anything there because .vercel is already ignored on link. Dropping both guard changes also removes the isVercelDeploymentEnv re-exports from @workflow/builders, which existed only for these two call sites, so the PR is now scoped to world selection.
| return; | ||
| } | ||
| const resolvedDataDir = path.resolve(dataDir); | ||
| if (resolvedDataDir.startsWith(`${tmpdir()}${path.sep}`)) { |
There was a problem hiding this comment.
Edge: startsWith(`` ${tmpdir()}${path.sep} ``) misses dataDir === tmpdir() exactly — on Vercel tmpdir() is /tmp, so dataDir: '/tmp' itself would warn spuriously. resolvedDataDir === tmpdir() || resolvedDataDir.startsWith(...) covers it.
There was a problem hiding this comment.
(AI) Fixed in 9d62ee2, and the warning moved to packages/world-local/src/build-target-mismatch.ts next to the shared advice string:
const resolvedDataDir = path.resolve(dataDir);
const tmp = tmpdir();
if (resolvedDataDir === tmp || resolvedDataDir.startsWith(`${tmp}${path.sep}`)) {
return;
}Covered by two cases in vercel-deployment-warning.test.ts: tmpdir() itself and a path under it stay quiet, and ${tmpdir()}-elsewhere still warns (so the prefix check can't be loosened back into a substring match).
| * deliberate choice rather than a misconfigured build. | ||
| */ | ||
| function warnIfRunningInVercelDeployment(dataDir: string): void { | ||
| if (warnedAboutVercelDeployment || !isVercelDeploymentEnv()) { |
There was a problem hiding this comment.
Coverage gap worth noting: this safety net keys on isVercelDeploymentEnv() at runtime, but in the scenario the new docs paragraph calls out (system environment variables disabled), the runtime env also lacks VERCEL/VERCEL_DEPLOYMENT_ID — so the warning stays silent exactly where it's needed.
A robust complement (fine as a follow-up): emit this same actionable message when the local world's first write fails with EROFS. That catches every misdetection variant regardless of env visibility, which env sniffing can't.
There was a problem hiding this comment.
(AI) Good catch, and it's more necessary than the comment assumes — done in this PR rather than as a follow-up.
Two findings while implementing it:
- world-local's existing
DataDirAccessErrornever fires in a deployment. It comes frominitDataDir(), which only runs fromstart(), andworld.start?.()is called only by@workflow/vitest— never by the Next.js integration. So a deployed app never hits that path. ensureDir()infs.tsswallowed every mkdir error, so EROFS surfaced later as ENOENT on the file, which is what the customer's failure looked like.
So the throw is in ensureDir() — the choke point every write goes through, env-independent as you suggested:
const code = (error as NodeJS.ErrnoException).code;
if (isUnwritableDirCode(code) && !(await isExistingDirectory(resolvedPath))) {
throw new UnwritableDataDirError(resolvedPath, code as string);
}
// Ignore if already existsUnwritableDataDirError extends WorkflowWorldError (so it maps to HTTP like other world errors) and carries the same advice string as the warning. Codes are EROFS/EACCES/EPERM; the existence re-check preserves the historical tolerance for races and for an unsearchable parent that stat can still resolve. Two tests in fs.test.ts cover both directions.
| const maybePort = workflows?.local?.port; | ||
| if (maybePort) { | ||
| process.env.PORT = maybePort.toString(); |
There was a problem hiding this comment.
Behavior change worth a line in the changeset: previously workflows.local.port set PORT whenever VERCEL_DEPLOYMENT_ID was absent, regardless of world. Keying it to the resolved local world is correct (PORT only feeds world-local's callback base URL — packages/world-local/src/config.ts:103), but since Next's dev server also honors PORT, anyone (mis)using this option to move their dev server port while targeting a non-local world silently loses that.
There was a problem hiding this comment.
(AI) Added to the changeset in 9d62ee2:
Select the Vercel World for builds that target Vercel without a deployment ID, such as
vercel buildfollowed byvercel deploy --prebuilt, and applyworkflows.local.portwhenever the Local World is the target.
Also dropped @workflow/builders from that changeset — after the two .gitignore reverts its only remaining change is a test.
| NODE_ENV: 'production', | ||
| }; | ||
|
|
||
| describe('isVercelDeploymentEnv', () => { |
There was a problem hiding this comment.
Test gap: nothing pins VERCEL: '1' with NODE_ENV unset (common outside framework processes — plain node scripts, CLI contexts), which should still resolve to Vercel. Cheap to add to this table.
There was a problem hiding this comment.
(AI) Added in 9d62ee2, derived from the existing fixture rather than a new literal so it can't drift:
test('detects a Vercel build that exposes no NODE_ENV', () => {
const { NODE_ENV: _unset, ...envWithoutNodeEnv } = PREBUILT_BUILD_ENV;
expect(isVercelDeploymentEnv(envWithoutNodeEnv)).toBe(true);
});… sides - Restore the `public/.well-known/workflow/v1/.gitignore` guard to `VERCEL_DEPLOYMENT_ID`, so prebuilt builds keep writing it. - world-vercel: name the cause and the fix when VERCEL_DEPLOYMENT_ID is missing, from the getDeploymentId() call that start() makes before any state write. - world-local: throw UnwritableDataDirError from ensureDir when the data directory cannot be created, instead of surfacing a later ENOENT. - Move the Vercel-deployment warning next to the shared advice string and exempt the temp dir by path equality, not prefix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
All review comments addressed in 9d62ee2 — and better than asked:
- The production-exposure concern was refuted with evidence:
start()awaitsworld.getDeploymentId()before any state write, and world-vercel throws whenVERCEL_DEPLOYMENT_IDis unset, ahead of any credential use. I verified both in the source. Turning the finding into the sharedmissingDeploymentIdMessage()(naming the build-time cause and theWORKFLOW_TARGET_WORLD=local+ rebuild fix) plus the mirror docs paragraph is the right response — the message was the real defect. - The
EROFS→UnwritableDataDirErrorchoke-point error inensureDir()closes the safety-net gap env sniffing can't, and the two pre-existing findings along the way (DataDirAccessErrornever firing in deployments; mkdir errors surfacing asENOENTon the file — exactly what the customer saw) make it clearly worth doing in this PR. - Both
.gitignoresites reverted to main, tmpdir edge fixed with prefix-sibling coverage, dead fallback replaced, changeset updated, NODE_ENV-unset test added.
Declining the world-vercel creation-time warning is fair given createWorld() runs during builds via getWorldHandlers(); the getDeploymentId() throw covers that direction at the only unambiguous moment. The build-time log line for the VERCEL=1 fallback path is fine as a follow-up.
Required checks are green. 
VaguelySerious
added
the
event-log-race-repro
Event Log Race Repro882 of 1400 latest repro runs hit event-log regressions. 1 run hit harness-side Run History
Latest Scenario Breakdown
Latest Non-Completed Runs
Showing 20 of 883 non-completed runs. |
|
No backport to This fixes a regression introduced by static build-time world injection, which is To override, re-run the Backport to stable workflow manually via |
#3144 landed after this branch was cut and adapted world selection to #2752's build-time model: `isVercelDeploymentEnv` treats `VERCEL=1` without a deployment ID as Vercel, because a `vercel build` on external CI had to decide which world to bake into the bundles. With selection back at runtime, `VERCEL_DEPLOYMENT_ID` is authoritative and always present inside a deployment, so the broader signal only changes the cases that are not deployments: a production server started locally or in CI with an env file from `vercel env pull` would resolve to the Vercel world and then fail for want of a deployment ID, where before #2752 it used the local world. The same applies to the local world's read-only-filesystem warning, which would fire in a writable one. The user-facing text from #3144 explains all of this in terms of build output and prescribes a rebuild, which no longer applies now that `WORKFLOW_TARGET_WORLD` takes effect on the next start. The rest of #3144 is independent of the injection mechanism and stays: `UnwritableDataDirError`, the warning itself, and the Vercel world's missing-deployment-ID message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Reported by users upgrading
5.0.0-beta.22→5.0.0-beta.36: the SDK used its local filesystem world inside a Vercel deployment when usingvercel build+vercel deploy --prebuiltin CI.Cause
Static world injection moved world selection from runtime to build time. The detection rule itself never changed —
VERCEL_DEPLOYMENT_ID ? vercel : local— only when it runs.VERCEL_DEPLOYMENT_IDdoes not exist whilevercel buildruns: no deployment has been created yet. Avercel pull --environment productionenv containsVERCEL=1,VERCEL_ENV,VERCEL_TARGET_ENV,VERCEL_URL,VERCEL_OIDC_TOKENand theVERCEL_GIT_*set, but no deployment ID. So a prebuilt build compiles@workflow/world-localinto the deployment,@workflow/nextadditionally pointsWORKFLOW_LOCAL_DATA_DIRat.next/workflow-data, and the deployment writes workflow state to a read-only filesystem.Nothing surfaced it. No guard warned when the resolved world was local on Vercel;
@workflow/world-localis always resolvable, so there was no missing-module error; andensureDir()in world-local swallowed everymkdirerror, so the read-only filesystem showed up later asENOENTon a state file rather than asEROFSon the data directory. Before static injection the same rule was evaluated per cold start, where the deployment ID does exist — hence the regression at beta.28 rather than a change in behavior anyone opted into.Change
Selection.
isVercelDeploymentEnv()in@workflow/utilsis now the single answer to "is this a Vercel deployment, or a build that becomes one", used by bothresolveWorkflowTargetWorldandgetWorldImport:VERCEL_DEPLOYMENT_IDpresent → Vercel (runtime, and Vercel's own build container).VERCEL=1→ Vercel, unlessVERCEL_ENV=development(vercel dev) orNODE_ENV=development(any framework dev server).VERCEL=1alone is not sufficient becausevercel env pullwrites it into.env.local, which frameworks load before the config runs — without the development exclusions a plainpnpm devwould flip to the Vercel world.workflows.local.portin@workflow/nextis keyed to the resolved local world instead of the deployment ID, since that option only feeds world-local's callback base URL. Note the behavior change: it previously setPORTwhenever no deployment ID was present, regardless of world.Both wrong-world directions now fail with an actionable message, so a misdetection names its cause instead of surfacing as an unrelated error:
start()awaitsworld.getDeploymentId()before writing any state, and world-vercel throws there whenVERCEL_DEPLOYMENT_IDis missing. That throw existed; its message named only the env var. One message inpackages/world-vercel/src/deployment-id.tsnow explains that the world was selected at build time and thatWORKFLOW_TARGET_WORLD=local+ a rebuild is the fix, shared bygetDeploymentId(),queue(), andresolveLatestDeploymentId().ensureDir()throwsUnwritableDataDirError(aWorkflowWorldError) when the data directory cannot be created withEROFS/EACCES/EPERMand does not already exist, instead of letting the caller's write fail asENOENT. This catches every misdetection variant, including one env sniffing cannot: a project with system environment variables disabled has noVERCEL*at runtime either. world-local also warns when it is created inside a Vercel deployment with a data directory outside the temp dir, which covers deployments built by an older SDK; the temp dir is exempt because it is writable on Vercel and therefore a deliberate choice.WORKFLOW_TARGET_WORLDcontinues to override in both directions, and the docs now describe build-time selection and both failure modes.Verification
Real
vercel build --prodagainst the linkednextjs-turbopackproject (fresh compile, not a cache hit), pulled production env, no deployment ID:grep -rl WORKFLOW_LOCAL_DATA_DIR .next/servergrep -rl 'world-vercel|WORKFLOW_VERCEL' .next/serverRunning that build output under a Vercel runtime env reproduces the customer's failure before the change (
POST /api/workflows/start→200,.workflow-data/{events,hooks,runs}written to disk) and after it selects the Vercel world instead.Serving that same build outside a deployment (the pulled-prod-env
next startcase raised in review) cannot reach production state: the run fails atgetDeploymentId()before any write and before credentials are used —500 VERCEL_DEPLOYMENT_ID environment variable is not set, no run created — which is the throw whose message this PR rewrote.Dev is unaffected: with
.env.localcontainingVERCEL=1/VERCEL_ENV=preview,pnpm devstill resolves the local world, starts a run successfully, and emits no warning.Trade-off
next build && next starton a developer machine whose.env.localcarries pulled production env now resolves the Vercel world rather than the local one, and fails at the firststart()with the message above. The alternative is leaving prebuilt pipelines broken by default;WORKFLOW_TARGET_WORLD=localplus a rebuild is the opt-out.🤖 Generated with Claude Code