Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- `--udid` with `--platform android` (and `--serial` with an Apple platform) now fails as the flag mistake it is — `INVALID_ARGS` naming the right flag — instead of reaching device resolution and answering `No Apple device with UDID emulator-5580` for an explicitly Android request. `--udid` addresses Apple devices, `--serial` addresses Android and HarmonyOS; matching pairs and requests that name no platform are unchanged.
- Breaking (`--session-lock strip`): a device selector that names a different device than the bound session is no longer silently discarded. `strip` exists to drop redundant platform/scope selectors; when it also dropped `--udid`/`--serial`/`--device`, the command kept running against the *bound* device instead of the one the caller named — a wrong-device action that looks like a success. Such a request now fails with `INVALID_ARGS` under both `reject` and `strip`, and the error carries the two identities structurally (`requestedDevice`, `boundDevice`) plus a hint offering the two real recoveries: close the bound session if the requested device is intended, or remove the selector if the bound device is. The hint no longer suggests `--session-lock strip` for an identity conflict, since following that advice is what produced the wrong-device run. Scope-only stripping (`--platform`, `--target`, `--ios-simulator-device-set`, `--android-device-allowlist`) is unchanged.
- iOS regular snapshots now apply one backend-neutral eligibility rule after every capture backend: a node survives when its accessibility type is interactive or it carries a non-empty label, identifier, or value. This removes the tree backend's extra "hittable non-Other" membership path and drops unlabeled decorative nodes consistently; labeled images, identifier-only nodes, and value-only nodes still survive. Raw snapshot membership is unchanged.
- Android `snapshot --raw` is now the acquired accessibility tree (#1832 C3): the three regular-projection pruners — nodes Android marks invisible, stale application windows, and covered same-window surfaces — no longer run at parse time, so `--raw` keeps everything the helper serialized (normalization only) and pruned content is recoverable for diagnosis. Regular and `-i` output is unchanged (property-checked identical to the previous implementation over 12,000 random tree × projection pairs, hidden-content hints included). Also: Android blocking-dialog recovery now reads the same daemon presentation an agent's `snapshot` sees instead of a hand-rolled subset, and acts on its occlusion result — a stale "App isn't responding" surface left under the foreground one no longer triggers recovery, and a covered "Close app" is never tapped ahead of the visible one; the Android freshness route signature no longer keys on `role`/`selected`, fields the Android backend never carries; and the Android helper's declared fidelity residues (no `checked`/`checkable`/`selected`/`long-clickable`, 5000-node cap before scoping, API-level cache-reset divergence) are recorded in `CONTEXT.md`.
- `agent-device mcp` now carries its own usage guidance, so MCP-only clients (Codex CLI, Cursor, custom agents) no longer depend on a separately installed skill (#1833). The handshake `instructions` — returned by both `server/discover` and, newly, the legacy `initialize` — is a compact (< 2 KB, the Claude Code truncation limit) workflow card: start with `open {app, foreground: true}` instead of probing, act with `settle: true` and continue from the diff, verify with `wait`/`is`/`get`/`find`, copy `@refs` byte-for-byte, recover from sparse/AX-unavailable, follow error hints, `close`. A new MCP-only `help` tool serves the full guides on demand: no `topic` returns the CLI's decision card; `topic` returns `agent-device help <topic|command>` verbatim (workflow, gestures, scripting, tv, macos, web, remote, debugging, …, or any tool name for its complete flag reference), prefixed with the one-line CLI→tool-property mapping. `help` is router-owned rather than a command descriptor, so it appears in `tools/list` only — not in the CLI, Node client, or `batch` — and its description tells the model it is not a startup step. Legacy `initialize` gains the optional `instructions` field; no other legacy field changes.
- Android `snapshot --scope` (and every selector command's `--scope`, e.g. `press "Save" --scope Panel`) now resolves scope exactly once, inside the Android projection, under the shared scope specification: the scope root is the first node **in document order** whose label, value, or identifier contains the scope text (case-insensitive) **and whose subtree still has content in the projection you asked for**, the result is that subtree re-rooted at depth 0, and no match returns an empty snapshot (#1832). That second clause is what makes `snapshot -i --scope panel` return the button inside a structural container `-i` drops, and stops a decorative heading that happens to match from emptying the snapshot. Before, Android ran two passes with contradictory rules — a breadth-first platform match that fell back to the full tree on a miss, then the daemon's document-order pass — so a shallower later container could win over an earlier match, and an interaction capture whose scope reached only the daemon layer was silently unscoped. `--depth` under `--scope` counts from the scope root, filtering the depths the response prints (a node shown at depth 0 is never hidden by `--depth 0`), and ancestor context above the scope root (a clickable row, a list) still shapes `-i` membership inside it. The rule is pinned by `contracts/fixtures/snapshot-scope-policy.json`, the same golden table the iOS runner will consume (#1797). Also new: `androidSnapshot.occlusionScanUnavailable: true` discloses an API 23 capture, where the helper cannot report `drawing-order` and covered same-window surfaces are therefore not pruned.
- New `hover <x y|@ref|selector>` command for `--platform web` (#1783). It moves the pointer over the target without pressing, so hover-gated UI — a message row's `...` toolbar, a menu that opens on pointer enter — becomes reachable through agent-device the way it already was through the underlying `agent-browser` backend (`mouse move`). It is a member of the targeted-touch family: same `@ref`/selector/coordinate targeting, occlusion and off-screen guards, and `--settle` (the settled diff carries the revealed controls with fresh refs, e.g. `+ @e4 [button] "Delete"`), but no `--verify`, since hover reveals rather than activates. `hover @ref` publishes as a portable selector line in recorded scripts, and the Node client exposes `interactions.hover`. Hover is a pointer state that touch platforms do not have, so `capabilities` advertises it on web only and iOS/Android/Linux reject it during admission with `UNSUPPORTED_OPERATION` and a hint naming `--platform web`; `longpress` remains the mobile hold-gesture verb.
Expand Down
15 changes: 15 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,21 @@ task touches:
typed `WebView` root and WebKit's `Other -> StaticText` wrapper pairs. It keeps raw diagnostics
unchanged, presents ordinary wrapper text as `StaticText`, and presents wrappers carrying WebKit's
numeric HTML heading level as `Heading`.
- Android snapshot projections (#1832 C3): the parsed helper tree is the acquired tree and is never
mutated. `--raw` presents it whole (normalization only). The regular projection additionally hides
what `collectAndroidHiddenNodes` classifies — nodes Android marks invisible to users, stale
application windows, and same-window surfaces a higher drawing-order sibling covers — and the
interactive projection applies membership on top. `interactive ⊆ regular ⊆ raw` holds by
construction; hidden-content hints and the scope root are derived per projection.
- Android declared residues (#1832): fidelity limits of the helper acquisition that no presentation
can repair, disclosed rather than papered over. (1) API 23 helper trees carry no `drawing-order`,
so the covered-surface rule cannot run — `androidSnapshot.occlusionScanUnavailable`. (2) The
helper serializes no `checked`/`checkable`/`selected`/`long-clickable`, so toggle and selection
state is invisible and nothing on Android may key on those fields (the freshness route signature
does not). (3) The helper caps at 5000 nodes before any projection, so a scoped target past the
cap vanishes with only the generic `truncated`. (4) Accessibility-cache reset differs by API level
(`clearCache()` on 34+, `setServiceInfo` re-apply below), an undisclosed acquisition-freshness
difference.
- AX-unavailable target invalidation: iOS/macOS runner behavior where a root accessibility snapshot
failure such as `kAXErrorIllegalArgument` marks the cached `XCUIApplication` target handle suspect.
The runner fails closed for degraded interactive snapshots, clears the cached target, and lets the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,10 @@ private static void clearAccessibilityCache(UiAutomation automation) {
// destination inside one AndroidComposeView. A persistent helper session reuses a single
// UiAutomation connection across captures, and its per-connection accessibility node cache is
// not always invalidated by such a swap. Drop the cache before each traversal.
//
// Declared residue (agent-device #1832): the reset mechanism differs by API level — the public
// clearCache() on 34+, a setServiceInfo() re-apply below — and the two are not known to be
// equivalent. This is an acquisition-freshness difference the payload does not disclose.
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
try {
automation.clearCache();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,9 @@ private static void appendTrueAttribute(StringBuilder xml, String name, boolean
}
}

// Declared residue (agent-device #1832): checked / checkable / selected / long-clickable are not
// serialized, so toggle and selection state is invisible to agents. Adding them is a helper
// protocol change (new attributes + host parser + fields on the wire node), tracked there.
private static void appendDrawingOrderAttribute(StringBuilder xml, AccessibilityNodeInfo node) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
appendAttribute(xml, "drawing-order", Integer.toString(node.getDrawingOrder()));
Expand Down
2 changes: 1 addition & 1 deletion src/__tests__/test-file-size-ratchet.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ const PINNED_TEST_FILE_LINES: Readonly<Record<string, number>> = Object.freeze({
'src/platforms/apple/core/__tests__/runner-session.test.ts': 2001,
'src/utils/__tests__/daemon-client.test.ts': 1910,
'src/utils/__tests__/output.test.ts': 1861,
'src/platforms/android/__tests__/snapshot.test.ts': 1660,
'src/platforms/android/__tests__/snapshot.test.ts': 1658,
'src/platforms/apple/core/__tests__/runner-client.test.ts': 1615,
'src/__tests__/client.test.ts': 1598,
'test/integration/provider-scenarios/android-lifecycle.test.ts': 1559,
Expand Down
114 changes: 114 additions & 0 deletions src/daemon/__tests__/android-system-dialog-occlusion.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
import { test, expect, vi } from 'vitest';

// #1832: blocking-dialog recovery reads the daemon presentation, so it must also consume that
// presentation's OCCLUSION result. A stale "App isn't responding" surface left under the foreground
// one must not trigger recovery, and a covered "Close app" must not be tapped ahead of the visible
// one — the exact disagreement routing through buildSnapshotState exists to remove.
vi.mock('../../platforms/android/snapshot.ts', () => ({ snapshotAndroid: vi.fn() }));
vi.mock('../../platforms/android/adb.ts', () => ({ runAndroidAdb: vi.fn() }));

import { snapshotAndroid } from '../../platforms/android/snapshot.ts';
import { runAndroidAdb } from '../../platforms/android/adb.ts';
import { recoverAndroidBlockingSystemDialog } from '../android-system-dialog.ts';
import { makeAndroidSession } from '../../__tests__/test-utils/session-factories.ts';
import { makeTestScreenRecordingResource } from '../../__tests__/test-utils/screen-recording-live-handle.ts';

const SCREEN = { x: 0, y: 0, width: 1080, height: 2400 };

/** A foreground surface drawn over everything below it — what makes the stale dialog unreachable. */
const foregroundSheet = (index: number) => ({
index,
type: 'com.google.android.material.bottomsheet.BottomSheetDialog',
identifier: 'com.example:id/bottom_sheet',
hittable: true,
rect: SCREEN,
});

function recordingSession(name: string) {
const session = makeAndroidSession(name);
session.screenRecording = makeTestScreenRecordingResource(session, {
backend: 'adb screenrecord',
outPath: '/tmp/anr.mp4',
startedAt: 0,
});
return session;
}

function tappedPoints(): Array<{ x: number; y: number }> {
return vi
.mocked(runAndroidAdb)
.mock.calls.filter((call) => (call[1] as string[]).includes('tap'))
.map((call) => {
const args = call[1] as string[];
const at = args.indexOf('tap');
return { x: Number(args[at + 1]), y: Number(args[at + 2]) };
});
}

test('recovery taps the reachable Close app, not a covered one that comes first', async () => {
vi.mocked(runAndroidAdb).mockReset();
// The live ANR dialog is on top; a stale "Close app" from an earlier one is still in the tree
// underneath the sheet. Document order puts the stale button FIRST, which is what the previous
// hand-rolled presentation selected: the first text match carrying a rect.
const staleCloseApp = {
index: 0,
type: 'Button',
label: 'Close app',
hittable: true,
rect: { x: 40, y: 300, width: 200, height: 80 },
};
const sheet = { ...foregroundSheet(1), rect: { x: 0, y: 0, width: 1080, height: 1000 } };
const dialog = {
index: 2,
type: 'TextView',
label: "App isn't responding",
rect: { x: 0, y: 1700, width: 1080, height: 60 },
};
const visibleCloseApp = {
index: 3,
type: 'Button',
label: 'Close app',
hittable: true,
rect: { x: 40, y: 1800, width: 200, height: 80 },
};
vi.mocked(snapshotAndroid)
.mockResolvedValueOnce({ nodes: [staleCloseApp, sheet, dialog, visibleCloseApp] } as never)
.mockResolvedValue({ nodes: [] } as never);
vi.mocked(runAndroidAdb).mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' } as never);

const result = await recoverAndroidBlockingSystemDialog({
session: recordingSession('anr-covered-button'),
});

expect(result.status).toBe('recovered');
expect(tappedPoints()).toEqual([{ x: 140, y: 1840 }]);
});

test('a fully covered dialog does not trigger recovery', async () => {
vi.mocked(runAndroidAdb).mockReset();
const staleDialog = {
index: 0,
type: 'TextView',
label: "App isn't responding",
hittable: true,
rect: { x: 0, y: 100, width: 1080, height: 60 },
};
const staleCloseApp = {
index: 1,
type: 'Button',
label: 'Close app',
hittable: true,
rect: { x: 40, y: 300, width: 200, height: 80 },
};
vi.mocked(snapshotAndroid).mockResolvedValue({
nodes: [staleDialog, staleCloseApp, foregroundSheet(2)],
} as never);
vi.mocked(runAndroidAdb).mockResolvedValue({ exitCode: 0, stdout: '', stderr: '' } as never);

const result = await recoverAndroidBlockingSystemDialog({
session: recordingSession('anr-covered-dialog'),
});

expect(result.status).toBe('absent');
expect(tappedPoints()).toEqual([]);
});
2 changes: 1 addition & 1 deletion src/daemon/__tests__/generic-settle.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { activateCompleteRefFrame } from '../ref-frame.ts';
import { setSessionSnapshot } from '../session-snapshot.ts';
import type { SessionStore } from '../session-store.ts';
import type { DaemonRequest, DaemonResponse, SessionState } from '../types.ts';
import { buildSnapshotState } from '../handlers/snapshot-capture.ts';
import { buildSnapshotState } from '../snapshot-state.ts';

// #1638 `--settle` on the GENERIC daemon route (scroll/back): the settled diff,
// its refs, and the ref-frame/generation dance are the same contract the touch
Expand Down
Loading
Loading