Consolidate API doc toggle, harden build memory, and fix OpenAPI client bundle#1490
Draft
gavinelder wants to merge 12 commits into
Draft
Consolidate API doc toggle, harden build memory, and fix OpenAPI client bundle#1490gavinelder wants to merge 12 commits into
gavinelder wants to merge 12 commits into
Conversation
Add a temporary diagnostic to the Netlify build command that logs the container's real cgroup memory cap (the value that triggers 137 OOM kills) alongside the misleading host view, so SSG concurrency/threads can be tuned against the true limit. Also consolidate API documentation behind a single EXCLUDE_PLATFORM_API lever: it now gates the API content plugin, the OpenAPI preset, and the llms-api plugin together (all read from platform-api-docs/docs). Adds an envFlag() helper so "false"/"0"/"" read as not-excluded instead of being truthy strings, and drops the now-redundant EXCLUDE_PLATFORM_OPENAPI var. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
✅ Deploy Preview for seqera-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The build runs on a Firecracker MicroVM with ~10 GiB dedicated RAM and 6 vCPUs (confirmed via the build-resource probe), so the previous inline single-process SSG left 5 cores idle and squeezed rendering into one 4 GB heap — the likely source of intermittent OOMs. Enable future.experimental_faster.ssgWorkerThreads and tune the pool via env vars: 4 workers, recycle past ~1.5 GB, task size 8. Peak ≈ 7-8 GB, under the VM's ~10 GiB, with the recycler freeing retained memory between tasks. Also corrects the stale netlify.toml comment (the old 137 kills came from the 32-thread default, not from limits being "unenforced"). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stop excluding platform-api, platform-cli, and wave from deploy-preview builds so previews exercise the full, heavy build — the real OOM stress case for the new recycling SSG worker pool. Building platform-api locally also makes the non-forced /platform-api/* proxy fall through to the built pages, so previews are self-contained rather than pulling that section from the production API site. Replace the start-only memory probe with internal/build-mem-probe.sh, which samples MemAvailable throughout the build and reports the peak pressure point — the metric that actually validates the build fits under the VM's ~10 GiB. Temporary; revert once validated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validation on PR #1490 confirmed a full-scope build OOMs (137) during Rspack bundling on the ~10 GiB build VM — well before SSG runs — so the platform-api/cli/wave exclusions are load-bearing, not optional. Restore the known-good lean preview scope (exclude API, CLI, Wave) and remove the temporary build-mem-probe.sh and its build.command override. Kept: the recycling SSG worker pool (helps the SSG phase of the in-scope build) and the EXCLUDE_PLATFORM_API consolidation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-enable full-scope preview build and set DISABLE_RSPACK_INCREMENTAL to test whether dropping Rspack's incremental-compilation state brings the full build under the ~10 GiB VM (it OOM'd at 137 during bundling without it). Re-add a finer 1s memory sampler that records peak used and the ramp tail, since the previous 3s sampler missed the spike. Temporary experiment on PR #1490 — revert scope/probe once measured. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The OpenAPI docs theme statically bundles postman-code-generators (all 35
language codegens) and postman-collection into the client bundle for the
API Explorer code snippets. These reference Node core modules (path, url,
…) that Webpack 4 auto-polyfilled but Rspack/Webpack 5 do not, so building
with platform-api included failed at client compile ("Can't resolve
'path'"). This is why the API docs build only on the older seqera-docs-api
toolchain and not the main Rspack site.
Add openApiNodePolyfillsPlugin: resolve.fallback maps path/url/buffer to
browser polyfills and stubs Node-only modules. Re-enable full scope on the
preview to test end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Full-scope validation on PR #1490 concluded: even with the OpenAPI Node polyfill fix and DISABLE_RSPACK_INCREMENTAL, a build including platform-api peaks at 100% of the ~10.7 GiB build VM and OOMs (137) during Rspack bundling. Building everything in one site isn't viable on the current VM. Restore the lean preview scope (exclude API/CLI/Wave) and remove the temporary probe and build.command override. Kept as latent/safe improvements: the OpenAPI polyfill plugin (inert when API is excluded; unblocks API builds on a larger VM or the API site's toolchain), DISABLE_RSPACK_INCREMENTAL (trims peak, no cold-build downside), and the recycling SSG worker pool. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gavinelder
changed the title
These browser polyfills back the OpenAPI client-bundle fix (openApiNodePolyfillsPlugin) but were only present transitively. Pin them as explicit dependencies so a future transitive bump can't silently drop them — the exact failure mode that broke the build in #1406, when postman-code-generators 2.1.1 removed its bundled `path` dependency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The changelog blog gate still used the raw `process.env.EXCLUDE_CHANGELOG ?` truthy check while every other EXCLUDE_/INCLUDE_ gate (and the diagnostic log) had moved to envFlag(). That left the "false"-is-truthy trap in one spot: EXCLUDE_CHANGELOG="false" would disable the changelog while the log reported it included. Use envFlag() so all flags read consistently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Trim build-config comments to state what the code does, not how it got
there: drop session/PR narrative ("confirmed on PR #1490", "measured",
"historical 137 kills", "now gates", "instead of inline"). Rationale that
explains a non-obvious choice (envFlag's "false"-is-truthy guard, the
OpenAPI Node polyfills) is kept; transition history belongs in the PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
justinegeffen
added
1. Editor review
Signed-off-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Member
|
This all makes sense to me. A single EXCLUDE_PLATFORM_API will help a lot. I'd check what's in the Netlify UI as (if I remember correctly) it might need to be updated as well to add the new variable at the same time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related build/config improvements for the docs site, plus a regression fix for the OpenAPI client bundle.
1. Fix OpenAPI client bundle (regression from #1406)
#1406 bumped
postman-code-generators2.1.0 → 2.1.1, which removedpathas a dependency (per its changelog). The OpenAPI docs theme statically bundlespostman-code-generators(all 35 language codegens) +postman-collectioninto the client bundle for the API Explorer code snippets, where theyrequire('path')(andurl). Webpack 4 auto-polyfilled Node core modules; Rspack/Webpack 5 do not — so once the userlandpathpackage was dropped, any build includingplatform-apifailed at client compile withCan't resolve 'path'.Fix:
openApiNodePolyfillsPluginaddsresolve.fallbackmappingpath/url/bufferto browser polyfills and stubbing Node-only modules. Inert whenplatform-apiis excluded.2. Consolidate API docs behind a single
EXCLUDE_PLATFORM_APIleverEXCLUDE_PLATFORM_APInow gates all three consumers ofplatform-api-docs/docstogether: the API content-docs plugin, the OpenAPI preset, and the (previously ungated)llms-apiplugin. Adds anenvFlag()helper so"false"/"0"/""read as not excluded — previously the bare truthiness check treated the string"false"astrue. Removes the now-redundantEXCLUDE_PLATFORM_OPENAPIvar (and its entry incheck-netlify-excludes.sh).3. Harden build memory
future.experimental_faster.ssgWorkerThreadsand tune via env (4 workers, recycle past ~1.5 GB, task size 8). Parallelizes SSG across the build VM's cores and bounds peak memory by recycling workers between tasks. Replaces the prior inertDOCUSAURUS_SSG_WORKER_THREAD_COUNT=1(which left SSG running inline in a single process).DISABLE_RSPACK_INCREMENTAL: drops Rspack incremental-compilation state to trim peak bundler memory (no downside on cold CI builds).Validation (build VM: 6 vCPU, ~10.7 GiB, Firecracker — no cgroup cap)
platform-apioffload toseqera-docs-api(via the non-forced/platform-api/*proxy) remains load-bearing. The polyfill fix unblocks API builds on the API site's toolchain / a larger VM.Follow-ups (not in this PR)
EXCLUDE_PLATFORM_OPENAPIenv var in the Netlify UI; confirm whetherEXCLUDE_PLATFORM_APIis also pinned there (UI overridesnetlify.toml).path-browserify/url/bufferfrom transitive to explicitpackage.jsondeps so a future bump can't drop them (exactly the chore: update dependency postman-code-generators to v2.1.1 #1406 failure mode).Related: #1406 (introduced the OpenAPI
pathregression this addresses)🤖 Generated with Claude Code