Skip to content

fix(mutation): move the repo size ratchet where the lane cannot reach it - #1977

Open
thymikee wants to merge 1 commit into
mainfrom
fix/1964-mutation-sandbox-repo-introspection
Open

fix(mutation): move the repo size ratchet where the lane cannot reach it#1977
thymikee wants to merge 1 commit into
mainfrom
fix/1964-mutation-sandbox-repo-introspection

Conversation

@thymikee

@thymikee thymikee commented Aug 22, 2026

Copy link
Copy Markdown
Member

Repairs the mutation baseline on main, as asked on #1964.

The defect

Stryker runs the suite from a sandbox copy under .tmp/stryker/. A test that asserts about the repository checkout itself — its files on disk, or its git history — therefore reads a repository that does not exist.

test-file-size-ratchet.test.ts is such a gate, and it fails there for two independent reasons:

  1. disableTypeChecks (Stryker's default) prepends // @ts-nocheck to every copied {test,src,lib}/**/*.ts, so all 26 pinned files read one line longer than they are.
  2. The sandbox has no origin/main, so the gate's history-backed half cannot resolve its merge-base.

Fixing either leaves the other — I confirmed that the hard way, by fixing (1) first and watching CI fail on (2). Its own .tmp skip entry cannot help either: that is matched relative to REPO_ROOT, which inside the sandbox is the sandbox.

The fix

Move the gate to scripts/__tests__/ and include it explicitly in unit-core — the address this repo already uses for maintained gates that are not src tests (the help-conformance, xctest-selection and package-closure gates all live there).

KERNEL_TEST_FILE_RE admits only root/package src tests, and its own comment already names scripts/__tests__ as unreachable by construction. So the gate leaves every mutation lane by virtue of where it lives: no classifier to recognise it, nothing to keep in sync, and no way for a future lane to pull it back in without deliberately widening that pattern.

Nothing about the gate's behavior changes. REPO_ROOT is ../.. from either address, and TEST_ROOTS already included scripts, so it measures exactly the same tree — 4/4 tests pass unmoved.

What this replaces, and why

The previous revision added a repoIntrospectionTestFiles scanner that classified tests by source text — a single-quoted walk-files import, or the string origin/main. That was the wrong boundary, as reviewed: a behavioral test could match it and be silently excluded, while an equivalent repo gate written with double quotes, a different walker, or a different base ref would be missed. The scanner, its test, and its justifying comment are all deleted; scripts/mutation/test-scope.ts is now byte-identical to main.

The one assertion added instead pins the invariant this fix depends on: isKernelTestFile accepts root/package src tests and rejects scripts/__tests__. That is the whole invariant, not a sample of it — widening the pattern would silently pull the gate back into every lane, and now fails loudly at that edit instead of confusingly during a dry run.

Verification

pnpm mutation:run --modules kernel-errors at this commit, .tmp cleared first:

  • scope 804 → 803 test files — exactly this gate
  • dry run clean; lane envelope pass at stage complete
  • score 74.8% (187 killed / 63 survived / 250, 0 timeouts) — unchanged from the pre-existing baseline, as expected: a file-length-and-history gate can never kill a kernel mutant
  • pnpm mutation:test 39/39 · pnpm check:layering 181/181 · typecheck, lint, format clean

stryker.config.json is untouched, so the config content hash the report and lane envelope carry is unchanged and scores stay comparable across this commit.

Whose fault this was

Nobody's, and in particular not #1964's. You and that PR's author independently measured 804 kernel-related test files on both the exact base and its head, with a zero-line diff between the lists. Before #1969 rewired the module graph, nothing pulled this gate into a lane's scope, so the trap stayed hidden.

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.37 MB 2.37 MB 0 B
JS gzip 794.8 kB 794.8 kB 0 B
npm tarball 916.5 kB 916.5 kB 0 B
npm unpacked 3.18 MB 3.18 MB 0 B

npm unpacked components

Component Base Current Diff
JS / dist source 2.51 MB 2.51 MB 0 B
Apple runner source/project 564.2 kB 564.2 kB 0 B
macOS helper source 54.5 kB 54.5 kB 0 B
Android helper artifacts 0 B 0 B 0 B
Other package files 44.5 kB 44.5 kB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 18.7 ms 18.7 ms +0.0 ms
CLI --help 51.3 ms 52.4 ms +1.1 ms

Top changed chunks: no changes in the largest emitted chunks.

Top changed packed files

No changed packed files.

@thymikee
thymikee force-pushed the fix/1964-mutation-sandbox-repo-introspection branch 2 times, most recently from f7b79e5 to c5d938d Compare August 22, 2026 15:18
@thymikee thymikee changed the title fix(mutation): stop Stryker rewriting the sandbox out of sync with the repo fix(mutation): exclude repo-shape gates from the mutation lane's scope Aug 22, 2026
@thymikee

Copy link
Copy Markdown
Member Author

The CI diagnosis is valid and every exact-head lane is green, but repoIntrospectionTestFiles is the wrong boundary. It classifies tests by source-text sniffing for one single-quoted walk-files import or origin/main: a behavioral test can match and be silently excluded, while an equivalent repo gate using double quotes, another walker, or another base ref is missed. The test proves only one current behavioral file is absent, not the claimed invariant.

Use the repository’s existing structural owner instead: move test-file-size-ratchet.test.ts under scripts/__tests__/ and explicitly include it in unit-core in vitest.config.ts. Mutation scope already admits only root/package src tests through isKernelTestFile, so the repo gate then becomes unreachable by construction and this scanner, speculative test, and paragraph-long justification can be deleted.

Stryker runs the suite from a sandbox copy under `.tmp/stryker/`, so a test that
asserts about the repository checkout itself — its files on disk, or its git
history — reads a repository that does not exist.

`test-file-size-ratchet.test.ts` is such a gate, and it fails there for two
independent reasons: `disableTypeChecks` (Stryker's default) prepends
`// @ts-nocheck` to every copied file, so all 26 pinned files read one line
longer than they are; and the sandbox has no `origin/main`, so the gate's
history-backed half cannot resolve its merge-base. Fixing either leaves the
other. Its own `.tmp` skip entry cannot help: that is matched relative to
`REPO_ROOT`, which inside the sandbox *is* the sandbox.

Move it to `scripts/__tests__/` and include it explicitly in `unit-core`, the
address the repo already uses for maintained gates that are not `src` tests.
`KERNEL_TEST_FILE_RE` admits only root/package `src` tests, and its comment
already names `scripts/__tests__` as unreachable by construction — so the gate
leaves every mutation lane by virtue of where it lives, with no classifier to
recognise it and nothing to keep in sync.

This replaces the source-text scanner of the previous revision, which was the
wrong boundary: it sniffed for a single-quoted `walk-files` import or the string
`origin/main`, so a behavioral test could match and be silently excluded while an
equivalent repo gate using double quotes, another walker, or another base ref
would be missed. The scanner, its test, and its justifying comment are all gone.

`REPO_ROOT` and the walked roots are unchanged — both addresses are two levels
below the repo root, and `TEST_ROOTS` already included `scripts`, so the gate
measures exactly what it did before.

The one new assertion pins the invariant this now depends on: `isKernelTestFile`
accepts root/package `src` tests and rejects `scripts/__tests__`. Widening that
pattern would silently pull the gate back into every lane.

Verified with `pnpm mutation:run --modules kernel-errors`: scope 804 -> 803 test
files, dry run clean, lane `pass` at stage complete, score 74.8%
(187 killed / 63 survived / 250) — unchanged. `pnpm mutation:test` 39/39,
`pnpm check:layering` 181/181, `typecheck`, `lint`, `format` clean.
`stryker.config.json` is untouched, so scores stay comparable.

Unblocks #1964, whose two mutation checks fail on main's tip without its code.
@thymikee
thymikee force-pushed the fix/1964-mutation-sandbox-repo-introspection branch from c5d938d to 8f7d9be Compare August 22, 2026 16:35
@thymikee thymikee changed the title fix(mutation): exclude repo-shape gates from the mutation lane's scope fix(mutation): move the repo size ratchet where the lane cannot reach it Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant