[SDK] Ship per-platform native binaries via optional-dependency packages (#512)#567
Open
MGudgin wants to merge 6 commits into
Open
[SDK] Ship per-platform native binaries via optional-dependency packages (#512)#567MGudgin wants to merge 6 commits into
MGudgin wants to merge 6 commits into
Conversation
This PR adds the per-platform npm package skeletons that will each ship only
their own host's native binaries, the first step toward making
`@microsoft/mxc-sdk` pull down just the consuming host's payload.
Details
* Add `sdk/platform-packages/<os>-<arch>/` for win32 x {x64,arm64},
linux x {x64,arm64}, and darwin-arm64 (five packages). Intel macOS
(darwin-x64) is intentionally omitted — it is not built in CI and no
Intel-Mac executor binary has ever shipped; local Intel-Mac dev still
resolves via the monorepo `src/target` fallback.
* Each `package.json` pins the meta version (0.7.0), declares `os`/`cpu`
(plus `libc: [glibc]` on Linux so musl/Alpine hosts don't install
glibc-linked binaries), and uses an explicit `files` allowlist of the exact
binaries it ships — not a catch-all glob — so stray artifacts (logs, .pdb,
.env, prior .tgz) are never published.
* Each manifest adds a `prepack` guard that refuses to pack when its primary
binary was not staged, preventing an empty package from being published if a
build silently fails.
* In-package layout is binaries-at-root, preserving the `bin/` + `snapshots/`
substructure the Windows micro-VM payload needs.
* A `platform-packages/.gitignore` allowlist tracks only the manifests and
READMEs; its comment documents that the publish boundary is the per-package
`files` list, not this ignore rule.
No behavior change yet: discovery and packaging still use `sdk/bin/`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the @microsoft/mxc-sdk distribution model so the meta package ships no native executors, and instead pulls exactly one host-matching per-platform package (@microsoft/mxc-sdk-<os>-<arch>) via optionalDependencies. It also updates runtime binary resolution to be fail-closed in production installs and updates build/CI/release validation to ensure platform packages are correctly staged and published.
Changes:
- Split native payload into five per-platform npm packages and pin them as
optionalDependenciesfrom the meta SDK package. - Update SDK binary discovery (
platform.ts/helper.ts) to trust onlyMXC_BIN_DIRor an identity/version-validated platform package in production. - Update build scripts, CI pipelines, tests, and docs to stage/pack/verify per-platform artifacts (including release-completeness gates).
Show a summary per file
| File | Description |
|---|---|
| tests/playground/src/main/main.ts | Updates microvm binary search path to the new sdk/platform-packages/<tuple> layout. |
| sdk/tests/unit/sandbox.test.ts | Adds unit tests for missing-binary diagnostics and executablePath escape hatch behavior. |
| sdk/tests/unit/proxy-availability.test.ts | Adds unit coverage for proxy availability decision logic. |
| sdk/tests/unit/postinstall.test.ts | Adds unit coverage ensuring postinstall tuple set stays in sync and warnings behave as intended. |
| sdk/tests/unit/platform.test.ts | Adds extensive unit coverage for per-platform package resolution, dev-vs-prod behavior, and fail-closed guarantees. |
| sdk/tests/unit/packaging.test.ts | Adds packaging-matrix tests binding on-disk platform packages to SUPPORTED_TUPLES and validating meta/package manifests. |
| sdk/tests/integration/windows-process-container.test.ts | Skips proxy E2E tests when proxy binaries are not available post-split. |
| sdk/tests/integration/test-helpers.ts | Refactors integration test helpers to separate shipped executors from test-only proxy binaries. |
| sdk/tests/integration/proxy-availability.ts | Introduces pure proxy-availability evaluation used by integration tests (and unit-tested). |
| sdk/tests/integration/package.test.ts | Updates integration packaging assertions for per-platform packages and ensures meta has no bin/. |
| sdk/tests/integration/linux-bubblewrap.test.ts | Adds proxy availability gating for Bubblewrap proxy E2E tests. |
| sdk/src/platform.ts | Implements per-platform package resolution, tuple support gating, dev/prod modes, and platform-package identity/version validation. |
| sdk/src/helper.ts | Improves missing-binary errors, and adjusts platform gating to honor explicit executablePath earlier. |
| sdk/scripts/postinstall-check.cjs | Adds non-failing postinstall warning logic for missing/unsupported platform package scenarios. |
| sdk/README.md | Documents per-platform binary delivery model and resolver precedence (dev vs production). |
| sdk/platform-packages/win32-x64/README.md | Adds platform-package readme (win32-x64). |
| sdk/platform-packages/win32-x64/package.json | Adds platform-package manifest with explicit files allowlist and prepack guard (win32-x64). |
| sdk/platform-packages/win32-arm64/README.md | Adds platform-package readme (win32-arm64). |
| sdk/platform-packages/win32-arm64/package.json | Adds platform-package manifest with explicit files allowlist and prepack guard (win32-arm64). |
| sdk/platform-packages/linux-x64/README.md | Adds platform-package readme (linux-x64). |
| sdk/platform-packages/linux-x64/package.json | Adds platform-package manifest (linux-x64) including libc: glibc. |
| sdk/platform-packages/linux-arm64/README.md | Adds platform-package readme (linux-arm64). |
| sdk/platform-packages/linux-arm64/package.json | Adds platform-package manifest (linux-arm64) including libc: glibc. |
| sdk/platform-packages/darwin-arm64/README.md | Adds platform-package readme (darwin-arm64). |
| sdk/platform-packages/darwin-arm64/package.json | Adds platform-package manifest (darwin-arm64). |
| sdk/platform-packages/.gitignore | Ensures only manifests/READMEs are tracked and binaries remain uncommitted. |
| sdk/package.json | Removes bin/ from shipped files, adds postinstall check, and introduces platform-package optional deps. |
| sdk/package-lock.json | Records optional dependency stubs and marks the package as having an install script. |
| scripts/sync-platform-package-versions.test.js | Adds unit tests for platform-package version/pin syncing. |
| scripts/sync-platform-package-versions.js | Adds tooling to stamp/check platform package versions and meta optionalDependency pins. |
| scripts/platform-package-payload.test.js | Adds unit tests for deriving staged payload file lists from platform manifests. |
| scripts/platform-package-payload.js | Adds helper to derive payload files from a platform package’s files allowlist. |
| scripts/check-release-completeness.test.js | Adds unit tests for release completeness checks (pins, tarballs, malformed manifests, payload staging). |
| scripts/check-release-completeness.js | Adds a canonical release-completeness gate for platform packages and produced tarballs. |
| README.md | Updates build output description to reflect staging into sdk/platform-packages/<tuple>/. |
| docs/macos-support/seatbelt-backend.md | Updates macOS dev binary discovery docs to the new platform-packages layout. |
| build.sh | Stages Linux executors into sdk/platform-packages/linux-<arch> and adds version-sync check/stamp. |
| build.bat | Stages Windows executors into sdk/platform-packages/win32-<arch> and adds version-sync check/stamp. |
| build-mac.sh | Stages macOS executor into sdk/platform-packages/darwin-<arch> (when shipped) and adds version-sync check/stamp. |
| .github/workflows/Versioning.Checks.Job.yml | Adds CI validation/tests for version-sync and release-completeness logic. |
| .github/workflows/SDK.Integration.Test.Job.yml | Updates CI to install meta + host platform tarballs explicitly and wires proxy artifacts for Linux. |
| .github/workflows/Package.NpmSdk.Job.yml | Packs meta and platform packages separately and adds a release-completeness verification step. |
| .github/workflows/Build.Windows.Job.yml | Stages shipped payload from platform manifest allowlist and uploads manifest-driven artifacts. |
| .github/copilot-instructions.md | Documents the new SDK packaging model, resolver behavior, and version-sync tooling. |
| .azure-pipelines/templates/SDK.Integration.Test.Job.yml | Updates ADO integration tests to install meta + platform tarballs and provide proxy artifacts on Linux. |
| .azure-pipelines/templates/Rust.Build.Job.yml | Adds manifest-driven staging for Windows payload to avoid silent omissions. |
| .azure-pipelines/templates/Package.NpmSdk.Job.yml | Updates ADO packaging to stage/pack per-platform packages and run release completeness gate. |
| .azure-pipelines/templates/1ES.Build.Stages.yml | Wires per-platform packaging parameters through the 1ES stage template. |
| .azure-pipelines/README.md | Documents the new platform-first/meta-last publishing flow and operational constraints. |
| .azure-pipelines/1ES.Release.yml | Publishes platform packages before the meta package in ESRP. |
Copilot's findings
Files not reviewed (1)
- sdk/package-lock.json: Generated file
- Files reviewed: 49/50 changed files
- Comments generated: 3
82cda72 to
1a22c92
Compare
This PR reworks binary discovery so the SDK first looks for the executor in the host's optional per-platform package, hardens the resolver into a sandbox-grade security boundary, and makes the (platform, arch) support model coherent. Details * Discovery resolves via a single `resolveExecutable` with an ESM-safe, lazily created `require` and identity/version validation of the resolved platform package (rejects a same-named package planted in an ancestor node_modules). * Fail-closed production resolution: in an installed (node_modules) layout the resolver trusts ONLY `MXC_BIN_DIR` and the validated platform package — it no longer falls through to `bin/<arch>` or `src/target` paths near node_modules where a planted binary could be executed as the sandbox. * Dev (monorepo) layout prefers freshly-built local binaries (`sdk/platform-packages`/`src/target`) over an installed registry package, so a local Rust build wins and build failures aren't masked. Mode is detected by the sibling Rust workspace (overridable in tests). * Supported-(platform, arch)-tuple model: `isSupportedPlatformTuple` replaces the arch-only check; `computeSupport` now reports `isSupported:false` for `darwin-x64` (Intel macOS) and other non-shipped tuples, and the missing-binary message no longer synthesizes a `darwin-x64` package name that 404s. `getSdkArch` no longer collapses unknown archs into `x64`. * Testability seams: `_setHostId` (drive non-host tuples), `_setDevMode`, `_setPlatformPackageDir`, and exported `_validatePlatformPackageDir` — so the Intel-Mac path, the validation guard, and fail-closed behavior are unit tested deterministically from any host. * Fixes two adjacent ESM bugs (`SDK_VERSION` and meta-root resolution used the CommonJS `require` global and silently failed); centralizes binary names in `getExecutableBinaryName`; skips `X_OK` on Windows `.exe`; `MXC_BIN_DIR` short-circuits. Tests * SDK unit suite green (189 pass / 4 skipped), incl. new cases: Intel-Mac unsupported, production fail-closed (empty/absent → null), MXC_BIN_DIR precedence in both modes, platform-package identity/version validation, and per-OS discovery (wxc/lxc/seatbelt). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This PR repoints the three build scripts to stage each host's binaries into its
per-platform package directory instead of the shared `sdk/bin/<arch>`, and adds
a hardened, tested version-sync step that keeps the platform packages locked to
the meta version.
Details
* Staging hygiene: each script now cleans the destination package dir (keeping
only the tracked `package.json` + `README.md`) before copying, so stale or
flag-toggled artifacts (e.g. nanvix files after toggling `--with-microvm`
off) never persist into the published tarball.
* Fail-fast: build.sh and build-mac.sh now `exit 1` if the primary executor
(lxc-exec / mxc-exec-mac) is missing, instead of `cp … || echo Warning`
which returned 0 and let an incomplete package ship.
* CI-aware stamping: under `CI` the scripts run the version sync in `--check`
mode (fail on drift) rather than mutating tracked files, so a build no longer
has a source-tree side effect that could mask drift.
* build.bat: the optional Windows binaries are now copied by iterating a list
instead of copy-pasted `if exist … copy` blocks, and the `npm install` /
`npm run build` chains gained `|| goto :error` so a TypeScript build
failure aborts the build instead of being swallowed.
* build-mac.sh only stages into shipped platform packages: darwin-x64 (Intel,
not a shipped package) is skipped — its binary stays in src/target for local
dev resolution.
* sync-platform-package-versions.js refactored into an exported, fixture-testable
`syncPlatformPackageVersions({ repoRoot, check })` (process.exit lives only in
the CLI wrapper). It is now two-pass (validate everything before any write, so
a failure cannot leave a torn/partially-stamped tree), restricts the scan to
`^(win32|linux|darwin)-(x64|arm64)$` (ignoring tooling/metadata dirs), and
validates that versions are well-formed semver. Adds a unit test
(scripts/sync-platform-package-versions.test.js, 8 cases).
Notes / sequencing
* The Linux build produces no NanVix payload (NanVix is a Windows backend; the
Linux package list builds only lxc-exec), so build.sh correctly stages only
lxc-exec — there are no Linux microVM artifacts to stage.
* Moving staging out of `sdk/bin` leaves the meta `files`/optionalDependencies
(phase4) and the integration-test install + resolution (phase5) to catch up;
those are handled in the later phases of this series.
Tests
* `node --test scripts/sync-platform-package-versions.test.js` — 8 pass.
* Verified the staging clean-step (shell `find` and batch `del`/`rmdir`)
preserves only package.json + README.md; `bash -n` clean on both shell
scripts; the `--check` CLI passes against the real tree (5 packages).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This PR flips the public install model: the meta `@microsoft/mxc-sdk` package ships no native binaries and instead pulls the host's payload from one of five per-platform `optionalDependencies`, so an install downloads only its own (OS, arch) bytes. Details * Remove `bin/` from the meta package `files` (tarball ships only `dist/` + the postinstall script + docs) and add the five `@microsoft/mxc-sdk-<os>-<arch>` packages as exact-pinned `optionalDependencies` (darwin-x64/Intel is not a shipped package). * Non-failing `postinstall` (scripts/postinstall-check.cjs): warns — never fails the install — when the host's platform package is absent, so a silent optional-dep skip (`--omit=optional`, offline mirror, transient registry error, or an as-yet-unpublished platform package) surfaces at install time instead of only at first spawn. Stays quiet when the package is installed or in the monorepo dev layout. * sync-platform-package-versions.js now reconciles the meta `optionalDependencies` against the on-disk platform packages as the single source of truth: `--check` fails on a MISSING pin, a stale/zombie pin with no backing package, a wrong version, or a non-exact range — not just value drift among existing keys; stamp mode creates/fixes/removes pins. Adds 6 unit tests for the pin logic (14 total) and wires the unit test into the Versioning Checks workflow alongside the `--check` gate. * Regenerate sdk/package-lock.json for the five optional deps (+ hasInstallScript). Sequencing notes (handled in phase5 release ops) * The lockfile's optional-dep entries are stubs without `resolved`/`integrity` because the platform packages are not published yet; the lockfile must be regenerated with integrity after the platform packages are first published, and the meta package must publish AFTER them (phase5). Tests * `npm run build` clean; `npm run test` 184 pass / 4 skipped; `node --test scripts/sync-platform-package-versions.test.js` 14 pass. * `npm pack --dry-run` ships 0 native binaries; `npm ci` succeeds and the postinstall is quiet in the monorepo; verified the postinstall warns when the platform package is genuinely absent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
48fbfb4 to
3af1957
Compare
…512) This PR changes the release pipeline to build, pack, and publish the five per-platform binary packages alongside the meta @microsoft/mxc-sdk package, and migrates the SDK packaging + integration lanes to GitHub Actions. Details * Release-completeness gate (scripts/check-release-completeness.js) is filesystem-canonical (on-disk sdk/platform-packages is the source of truth): rejects an empty set, requires the meta to pin exactly those packages at their versions (no missing/wrong/zombie optionalDependencies), requires every expected platform tarball, and rejects ANY stray .tgz. The stray scan is now RECURSIVE (round-3 P1-2) so a nested tarball under the published folder can't bypass the guard. * New scripts/platform-package-payload.js derives a platform package's shipped build-artifact list from its manifest `files` allowlist (single source of truth). The Windows build job uses it to STAGE the shipped payload — including the micro-VM nanvixd.exe / kernel snapshots and wslcsdk.dll — preserving the bin/ + snapshots/ substructure, and HARD-FAILS if any allowlisted artifact is missing (round-3 P0-1 / P1-3). The hardcoded short upload+verify list is gone, so a payload entry can never be silently dropped from the artifact again. * Proxy-E2E enforcement (MXC_REQUIRE_PROXY_TESTS) is now default-ON for EVERY integration lane (job-level in GHA, all per-target blocks in ADO) rather than Linux-only (round-3 P0-2). A lane that runs the sandbox but forgets to wire the proxy now hard-fails instead of skipping green. * Both pipeline callers pass PLATFORM_PACKAGES_DIR to the gate. Pushback / conscious deferrals (documented in-code) * Windows proxy E2E still skips on the hosted runner — not a regression. The public GHA/ADO Windows runner has no runnable AppContainer/BaseContainer sandbox (MXC_SKIP_OS_BUILD_DEPENDENT_TESTS), so `sandboxSkipReason` short-circuits the proxy describe BEFORE the availability check. Setting MXC_REQUIRE_PROXY_TESTS job-wide is the honest fix: it makes the requirement uniform and self-enforcing for any lane that CAN run the sandbox, without faking coverage Windows can't provide. * arm64 integration lanes (win32-arm64 / linux-arm64) are deferred with an explicit matrix comment: the arm64 sandbox paths are unverified on the hosted arm64 runners and enabling them needs runner-side validation not done here. Conscious deferral, not an accidental gap. Tests * New scripts/platform-package-payload.test.js (4) + recursive nested-stray-tgz case added to check-release-completeness.test.js. All script tests pass (29 total across gate + payload + version-sync). * Workflow YAML validated (Build.Windows, GHA + ADO integration templates parse). * SDK build clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This PR is the tests + docs capstone for the per-platform packaging model: it adds packaging-invariant and discovery tests and aligns the docs. Details * packaging.test.ts derives the platform-package set from disk (single source of truth, so a removed package like darwin-x64 can't pass a stale hardcoded matrix) and asserts, per package: name/version/os/cpu, linux `libc:[glibc]`, an explicit `files` allowlist (no `**/*`), a `prepack` guard, and `repository.directory`. It also asserts the meta `optionalDependencies` EXACTLY pin the on-disk packages (no missing, no zombie, exact versions), runs `npm pack --dry-run` to prove the meta tarball ships zero native binaries, and checks that only `package.json`/`README.md` are git-tracked under platform-packages. Paths resolve from the test file (not `process.cwd()`) so the suite can't crash the runner from another directory. * platform.test.ts gains macOS coverage (`findSeatbeltExecutable` preferred / fallback / MXC_BIN_DIR), and its temp-dir cleanup now tolerates per-dir failures so one rmSync throw can't leak the rest. * `test:unit` switched to directory-glob discovery so a new `*.test.js` is picked up automatically instead of needing a manual allowlist edit. * Docs corrected: copilot-instructions now states the real resolution precedence (`MXC_BIN_DIR` first/override, then platform package, legacy bin/, dev output) and the five-package model; the root README and the seatbelt guide no longer point at the retired `sdk/bin/<arch>` path (`findDarwinExecutable` → `findSeatbeltExecutable`); sdk/README is five-package. Pushback / already-resolved * The version-sync script IS unit-tested — phase3 added `scripts/sync-platform-package-versions.test.js` (extended to 14 cases in phase4) and CI runs it; the review's "still untested" note reflects the pre-revision series. * The `_setPlatformPackageDir` test seam is kept (it mirrors the existing `_setProbeRunner` / `_setWindowsBuildQuery` seams in the same module); the cross-file-leak risk is mitigated by robust afterEach reset. * darwin-x64 false-confidence is gone: the suite validates only the five on-disk packages. Tests * `npm run build` clean; `npm run test` 226 pass / 7 skipped (incl. the new npm-pack, git-hygiene, exact-pin, and macOS-discovery cases). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3af1957 to
2638e49
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
@microsoft/mxc-sdkinstall only the host platform's native binaries bysplitting the executor payload into five per-platform packages
(
@microsoft/mxc-sdk-<os>-<arch>) referenced as exact-pinned, os/cpu-filteredoptionalDependenciesof a binary-free meta package. An install pulls just thematching host's payload instead of every platform's binaries.
This PR is presented as six reviewable commits (scaffold → discovery → build
scripts → meta/manifest → pipeline → tests/docs) but is intended to land as a
single squash-merge: the phases are a review-time decomposition, not
independently shippable units — dropping the bundled
bin/foroptionalDependencies(phase 4) is only safe alongside the publishing pipeline(phase 5), so the series must merge atomically to avoid a binary-less
main.Closes #512
Details
win32-x64,win32-arm64,linux-x64,linux-arm64,darwin-arm64. Intel macOS (darwin-x64) is intentionally notshipped. The
win32-arm64manifest ships only the arch-independent payload —the micro-VM payload (
nanvixd+ kernel assets) is x64-only.platform.ts): in an installed tree the executoris sourced ONLY from
MXC_BIN_DIRor the identity/version-validated installedplatform package — planted binaries in legacy
bin/<arch>orsrc/targetpaths are never executed.
SUPPORTED_TUPLESis the single source of truth. Anexplicit
executablePathis honored before the unsupported-platform gate (boththe low-level resolver and the public
resolveExecutableAndArgs).fail-fast on a missing mandatory component. The Windows CI build derives the
shipped payload from the manifest
filesallowlist(
scripts/platform-package-payload.js), stages it preserving thebin/+snapshots/substructure from an absolute workspace-rooted target path,and hard-fails on a missing artifact. The official ADO Windows build stages the
same manifest payload (manifest-driven, node-free).
bin/, gains the fiveoptionalDependencies,and ships a non-failing
postinstallthat warns when the host's platformpackage is absent OR installed without its native binary.
sync-platform-package-versions.jskeeps manifests + pins in lockstep(CI
--check).the meta package. The filesystem-canonical release-completeness gate treats
sdk/platform-packagesas the source of truth: rejects an empty set, requiresthe meta to pin exactly those packages at their versions, requires every
expected tarball, verifies every manifest
filespayload entry is actuallystaged on disk (so
npm packcannot silently omit a binary), hard-fails ona malformed platform manifest, and rejects ANY stray
.tgz(recursive scan).sdk/README.md,copilot-instructions.md, playground) describe theper-platform delivery model and the production-vs-dev resolver order.
Tests
fail-closed paths, packaging matrix bound bidirectionally to
SUPPORTED_TUPLES,micro-VM x64-only payload assertions, postinstall warn/skip/unsupported +
installed-but-binary-less warning +
runPostinstallCLI seam, missing-binaryerror names the optional package, explicit-
executablePathescape hatch on anunsupported host (both resolver entry points).
.tgz, on-disk payloadverification, malformed-manifest hard-fail, payload helper, version-sync.
sync --checkOK.tsc --noEmitclean.Reviewer notes
5 non-blocking, all resolved; 1 deferred as a non-blocking follow-up).
Rust.Build.Job.ymlmanifest-drivenpayload-staging step (ESRP/agent environment is pipeline-only). Please confirm
on a pipeline run that the
win32-x64micro-VM payload is staged into theartifact. The on-disk release-completeness gate is the safety net — a deficient
package now fails the release loudly rather than publishing silently.
for the glibc-only Linux package npm skips; a robust fix needs libc detection.
Non-blocking, no regression on glibc hosts.
Microsoft Reviewers: Open in CodeFlow