Fence browser observations by generation

[rgarcia]

summary

• collect browser AX trees into a structured, generation-fenced observation
• retry collection when target metadata or frame generations change
• defer presentation and ref minting until the observation is stable
• route browser snapshot and find through the same observation

stack

1. this PR — observation core
2. Add semantic browser waits #64 — semantic waits and frame hardening
3. Add verified browser action plans #65 — verified dependent action plans

size

542 changed lines: 388 additions, 154 deletions.

testing

• npm run typecheck
• npm test --workspace @onkernel/cua-agent — 170 passed, 16 live tests skipped
• git diff --check origin/main...HEAD

[rgarcia]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Missing TSDoc on observation exports
  • Added TSDoc comments to all newly exported observation interfaces, the ObservationChangedError class, and cohortKey to satisfy documentation requirements.

Or push these changes by commenting:

@cursor push 0ee770931c

Preview (0ee770931c)

diff --git a/packages/agent/src/translator/browser-observation.ts b/packages/agent/src/translator/browser-observation.ts
--- a/packages/agent/src/translator/browser-observation.ts
+++ b/packages/agent/src/translator/browser-observation.ts
@@ -1,3 +1,4 @@
+/** Minimal accessibility node shape consumed by browser observation rendering. */
 export interface AXNode {
 	nodeId: string;
 	ignored?: boolean;
@@ -10,11 +11,13 @@
 	childIds?: string[];
 }
 
+/** Indexes node ref ordinals within role/name cohorts for stable references. */
 export interface NthIndex {
 	index: Map<string, number>;
 	cohorts: Map<string, number>;
 }
 
+/** Carries frame/session metadata needed to render and trace an observed node. */
 export interface RenderContext {
 	targetId: string;
 	frameKey: string;
@@ -25,23 +28,27 @@
 	cursorIds?: ReadonlySet<number>;
 }
 
+/** One rendered observation line plus source node and render context. */
 export interface ObservationLine {
 	text: string;
 	refNode?: AXNode;
 	ctx: RenderContext;
 }
 
+/** Stitched frame tree with node lookup and per-frame render context. */
 export interface FrameStitch {
 	byId: Map<string, AXNode>;
 	roots: string[];
 	ctx: RenderContext;
 }
 
+/** Flattened observed node entry paired with the context it came from. */
 export interface ObservedNode {
 	node: AXNode;
 	ctx: RenderContext;
 }
 
+/** Full browser observation snapshot assembled from stitched frame trees. */
 export interface BrowserObservation {
 	targetId: string;
 	tree: FrameStitch;
@@ -52,6 +59,7 @@
 	generations: Map<string, number>;
 }
 
+/** Cached presentation artifact derived from a browser observation snapshot. */
 export interface BrowserPresentation {
 	observation: BrowserObservation;
 	cacheKey: string;
@@ -59,6 +67,7 @@
 	shape: string;
 }
 
+/** Signals that observation data changed mid-collection and must be retried. */
 export class ObservationChangedError extends Error {
 	constructor(message = "Browser observation changed during collection") {
 		super(message);
@@ -80,6 +89,7 @@
 	return { index, cohorts };
 }
 
+/** Builds a stable cohort identifier from a node's role/name pair. */
 export function cohortKey(role: string, name: string): string {
 	return `${role}\u0000${name}`;
 }

_{You can send follow-ups to the cloud agent here.}

[rgarcia]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0c29b34. Configure here.}