Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions .github/workflows/ios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,36 @@ on:
- '.github/workflows/deploy.yml'
- '.github/workflows/pr-preview.yml'
- '.github/workflows/pr-preview-cleanup.yml'
# Routing (#1781 A9-2). Everything below is derived from the affected selector's
# device-lane ownership (scripts/check-affected/device-lanes.ts): unit tests no device
# lane runs, and trees another platform family owns. `pnpm check:gate-manifest` asserts
# this list against the selector over every tracked path, both ways — a path the selector
# routes to the iOS lane (or fails open on: tooling, workflows, package manifests) must not
# appear here, and a path it classifies as another family's or as a unit test must. The
# `push` trigger has no filter: main runs every lane unconditionally.
- '**/*.md'
- 'src/**/*.test.ts'
- 'src/**/__tests__/**/*.ts'
- 'packages/*/src/**/*.test.ts'
- 'packages/*/src/**/__tests__/**/*.ts'
- 'src/platforms/android/**'
- 'src/platforms/harmonyos/**'
- 'src/platforms/linux/**'
- 'src/platforms/vega/**'
- 'src/platforms/web/**'
- 'packages/platform-android/src/**'
- 'packages/platform-harmonyos/src/**'
- 'packages/platform-linux/src/**'
- 'packages/platform-vega/src/**'
- 'packages/platform-web/src/**'
- 'android/**'
- 'linux/**'
- 'test/integration/replays/android/**'
- 'test/integration/replays/linux/**'
- 'test/integration/replays/macos/**'
- 'test/integration/android-**'
- 'test/integration/smoke-android-**'
- 'test/integration/smoke-web-**'
push:
branches:
- main
Expand Down
32 changes: 28 additions & 4 deletions docs/agents/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,19 @@ hand-maintained path map:
script fails loudly instead of dropping a gate.
- A **small explicit build-ownership layer** covers the paths whose owning build
cannot be derived: Swift runner, Android helpers, macOS helper, MCP metadata,
and the public package surface (itself derived from `package.json` `exports`).
the TS/Swift golden tables (`contracts/fixtures/`), and the public package
surface (itself derived from `package.json` `exports`).
- **Device lanes** (`replay-ios`, `replay-ios-device`, `replay-macos`,
`replay-android`, `replay-linux`, `web-smoke`) are owned by platform family
(`scripts/check-affected/device-lanes.ts`): a path under a family-tagged tree
(`packages/platform-<family>/`, `src/platforms/<family>/`, `android/`,
`test/integration/replays/<leaf>/`, the lane-prefixed `test/integration/`
smoke files) owns that family's lanes; untagged runtime surface owns every
lane; unit tests under `src/` and `packages/*/src/` own none. The tags are
directory-level only — `src/daemon/android-system-dialog.ts` is a naming
convention, not a boundary, and stays shared. `ios.yml`'s `pull_request`
`paths-ignore` is routed on this ownership and held to it both ways by the
gate manifest (below); `push` to main runs every lane unconditionally.

Changed-file discovery folds working-tree state into the local plan: in the
default local mode (`--head HEAD`) it unions the committed `base..HEAD` diff with
Expand Down Expand Up @@ -284,9 +296,21 @@ step executes on every run. Raw shell can still run project code, but it cannot
manifest. The check proves the smaller structural claim that every registered gate has an explicit
CI owner and every affected path can reach one.

The three facts the manifest cannot derive live together in `scripts/gate/declarations.ts`: one
coverage wrapper, one reporting-only `test:*` script, and the Android replay owner hidden inside a
third-party action's `script:` input.
The facts the manifest cannot derive live together in `scripts/gate/declarations.ts`: opaque
runners, reporting-only `test:*` scripts, unprovable and manual-only owners, and the **routed
lanes** — a `pull_request` lane whose `paths-ignore` list is asserted against the selector over
every tracked path, both ways: a path the selector fails open on or routes to one of the lane's
declared or sampled checks must start the lane, and a path it classifies as another family's
device-lane surface or as a unit test must not (`scripts/gate/routing.ts`). GitHub evaluates
`paths-ignore` before a runner is allocated, so this is routing with no job on the critical
path; the assertion is what keeps the hand-written glob list a derived artifact.

Two limits of the mechanism, both inherent to `paths-ignore` rather than to the assertion:
GitHub's path filters examine only the **first 300 changed files**, so a PR larger than that can
skip a routed lane on the strength of its first 300 paths alone (`push` to `main` has no filter
and is the backstop); and a lane may name a *sibling* workflow file exactly to say it does not
use it, but never a file in its own `uses:` closure — the composite actions its steps run, plus
its own definition — which the assertion refuses.

## Mutation report over decision kernels

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@
"fallow:baseline": "fallow dead-code --save-baseline fallow-baselines/dead-code.json --summary && fallow health --report-only --save-baseline fallow-baselines/health.json --summary",
"check:fallow": "fallow audit",
"check:affected": "node --experimental-strip-types scripts/check-affected/run.ts",
"check:affected:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/check-affected/model.test.ts scripts/check-affected/platform-packages.test.ts scripts/check-affected/run.test.ts",
"check:affected:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/check-affected/model.test.ts scripts/check-affected/platform-packages.test.ts scripts/check-affected/device-lanes.test.ts scripts/check-affected/run.test.ts",
"gate": "node --experimental-strip-types scripts/gate/run.ts",
"check:gate-manifest": "node --experimental-strip-types scripts/gate/check.ts",
"check:gate-manifest:test": "node --experimental-strip-types scripts/node-test-tmpdir.ts --experimental-strip-types --test scripts/gate/*.test.ts",
Expand Down
178 changes: 178 additions & 0 deletions scripts/check-affected/device-lanes.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
// Device-lane ownership (#1781 A9-2): planted paths per family, and the two properties the
// ios.yml routing leans on — an Apple-owned change carries its iOS checks in a narrow plan, and
// a tooling change beside it still fails open to the full set.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { deviceLaneLeaf, deviceLanesFor, isDeviceLaneSurface, isUnitTest } from './device-lanes.ts';
import { selectChecks, type CheckId } from './model.ts';

const IOS: readonly CheckId[] = ['replay-ios', 'replay-ios-device'];

function lanes(file: string): CheckId[] {
const plan = selectChecks({ changedFiles: [file] });
assert.equal(plan.failOpen, false, `${file} must not fail open`);
return plan.checks.filter((id) =>
/^replay-|^web-smoke$|^swift-runner-|^macos-helper$|^android-helpers$/.test(id),
);
}

test('a TypeScript-only Apple change selects the iOS and macOS lanes without a Swift build', () => {
for (const file of [
'src/platforms/apple/core/apps.ts',
'packages/platform-apple/src/runtime.ts',
'src/platforms/apple/os/macos/desktop.ts',
]) {
assert.deepEqual(lanes(file), ['replay-ios', 'replay-ios-device', 'replay-macos'], file);
assert.ok(
selectChecks({ changedFiles: [file] }).reasons.some(
(reason) => reason.rule === 'own:device-lane:apple',
),
);
}
assert.deepEqual(lanes('src/daemon/snapshot-presentation/ios/action-shelf.ts'), [...IOS]);
assert.deepEqual(lanes('test/integration/replays/macos/01-desktop.ad'), ['replay-macos']);
});

test('a Swift runner change selects both builds and every Apple lane', () => {
assert.deepEqual(lanes('apple/runner/AgentDeviceRunner/Sources/Foo.swift'), [
'swift-runner-ios',
'swift-runner-macos',
'replay-ios',
'replay-ios-device',
'replay-macos',
]);
});

test('an iOS replay script and the iOS smoke files own the iOS lanes', () => {
for (const file of [
'test/integration/replays/ios/simulator/01-settings.ad',
'test/integration/replays/ios/fixture/02-checkout-release.ad',
'test/integration/smoke-ios-simulator.test.ts',
'test/integration/ios-simulator-e2e/live-runner.ts',
'test/integration/ios-simulator-e2e-cleanup.test.ts',
]) {
assert.deepEqual(lanes(file), [...IOS], file);
}
});

test('shared runtime surface owns every device lane', () => {
for (const file of [
'src/daemon/handlers/session.ts',
'packages/kernel/src/errors.ts',
'src/daemon/android-system-dialog.ts', // naming convention in a shared dir, not a boundary
'test/integration/smoke-daemon-clean.test.ts',
'src/platforms/install-source.ts',
]) {
assert.equal(deviceLaneLeaf(file), 'shared', file);
assert.deepEqual(lanes(file), [
'web-smoke',
'replay-ios',
'replay-ios-device',
'replay-macos',
'replay-linux',
'replay-android',
]);
}
});

test('another family owns only its own lanes, so an Android-only change carries no iOS check', () => {
assert.deepEqual(lanes('src/platforms/android/perf.ts'), ['replay-android']);
assert.deepEqual(lanes('packages/platform-android/src/inventory.ts'), ['replay-android']);
assert.deepEqual(lanes('android/snapshot-helper/src/main/java/X.java'), [
'android-helpers',
'replay-android',
]);
assert.deepEqual(lanes('test/integration/replays/android/emulator/01-settings.ad'), [
'replay-android',
]);
assert.deepEqual(lanes('test/integration/smoke-android-emulator.test.ts'), ['replay-android']);
assert.deepEqual(lanes('test/integration/android-emulator-e2e/live-runner.ts'), [
'replay-android',
]);
assert.deepEqual(lanes('src/platforms/linux/snapshot.ts'), ['replay-linux']);
assert.deepEqual(lanes('linux/atspi-dump.py'), ['replay-linux']);
assert.deepEqual(lanes('src/platforms/web/provider.ts'), ['web-smoke']);
assert.deepEqual(lanes('test/integration/smoke-web-platform.test.ts'), ['web-smoke']);
// Families with no CI lane fall through to the static gates only.
assert.deepEqual(lanes('src/platforms/harmonyos/hdc.ts'), []);
assert.deepEqual(lanes('packages/platform-vega/src/index.ts'), []);
});

test('the fixture app owns the mobile lanes whichever subtree changes', () => {
assert.deepEqual(lanes('examples/test-app/app/index.tsx'), [
'replay-ios',
'replay-ios-device',
'replay-android',
]);
assert.equal(deviceLaneLeaf('examples/test-app/modules/lab/android/build.gradle'), 'fixture-app');
});

test('unit tests under src/ and packages/*/src own no lane', () => {
for (const file of [
'src/daemon/selectors.test.ts',
'src/__tests__/contracts/interaction-guarantees.test.ts',
'src/platforms/apple/__tests__/apps.test.ts',
'packages/platform-apple/src/runtime.test.ts',
]) {
assert.ok(isUnitTest(file), file);
assert.ok(!isDeviceLaneSurface(file), file);
assert.deepEqual(lanes(file), [], file);
}
// Non-TypeScript under __tests__ is a fixture the selector cannot place, not a unit test.
assert.ok(!isUnitTest('src/__tests__/test-utils/android-ime-capture.raw.json'));
// Package tests outside src/ have no owner and keep failing open.
assert.ok(!isUnitTest('packages/maestro/test/conformance/verify.test.ts'));
assert.equal(
selectChecks({ changedFiles: ['packages/maestro/test/conformance/verify.test.ts'] }).failOpen,
true,
);
});

test('a mixed tree is shared, and two Apple leaves are still Apple', () => {
assert.equal(deviceLaneLeaf('src/platforms/apple/os/macos/x.ts'), 'apple');
assert.equal(deviceLaneLeaf('test/integration/android-emulator-e2e/ios/x.ts'), 'shared');
assert.deepEqual(deviceLanesFor('src/platforms/apple/os/macos/x.ts').lanes, [
'replay-ios',
'replay-ios-device',
'replay-macos',
]);
});

test('the surface never narrows a file the selector could not otherwise place', () => {
for (const file of [
'src/global.d.ts',
'src/foo.json',
'test/integration/ios-simulator-e2e/a.png',
]) {
assert.ok(!isDeviceLaneSurface(file), file);
assert.equal(selectChecks({ changedFiles: [file] }).failOpen, true, file);
}
});

test('a golden table owns the parity unit test and both runner builds', () => {
const plan = selectChecks({ changedFiles: ['contracts/fixtures/tap-point-policy.json'] });
assert.equal(plan.failOpen, false);
for (const id of ['unit', 'swift-runner-ios', 'swift-runner-macos'] as const) {
assert.ok(plan.checks.includes(id), id);
}
assert.ok(plan.reasons.some((reason) => reason.rule === 'own:golden-table'));
});

test('a tooling change beside an Android-only change still fails open to the full set', () => {
for (const tooling of [
'.github/workflows/ios.yml',
'.github/actions/setup-apple-runner-build/action.yml',
'package.json',
'pnpm-lock.yaml',
'tsconfig.json',
'vitest.config.ts',
'scripts/check-affected/device-lanes.ts',
'packages/platform-android/package.json',
]) {
const plan = selectChecks({ changedFiles: ['src/platforms/android/perf.ts', tooling] });
assert.equal(plan.failOpen, true, tooling);
assert.ok(plan.checks.includes('swift-runner-ios'), tooling);
assert.ok(plan.checks.includes('replay-ios'), tooling);
}
});
Loading
Loading