Add verified browser action plans#65
Conversation
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2d43984. Configure here.
092e9a6 to
f187fc1
Compare
2d43984 to
32495d9
Compare
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Batch summary ignores failed waits
- The batch status now always surfaces failed browser waits before action-plan outcomes, so mixed batches no longer hide wait failures.
- ✅ Fixed: Missing TSDoc on exported formatter
- Added TSDoc to
formatBrowserActResultdescribing its purpose, inputs, and why diff output is truncated versus structured details.
- Added TSDoc to
Or push these changes by commenting:
@cursor push 33c635dbb5
Preview (33c635dbb5)
diff --git a/packages/agent/src/tools.ts b/packages/agent/src/tools.ts
--- a/packages/agent/src/tools.ts
+++ b/packages/agent/src/tools.ts
@@ -199,17 +199,15 @@
const acts = readResults.flatMap((read) => read.type === "browser_act" ? [read.result] : []);
const waits = readResults.flatMap((read) => read.type === "browser_wait_for" ? [read.result] : []);
const failedWait = ["interrupted", "timed_out", "unverifiable"].find((status) => waits.some((wait) => wait.status === status));
- const statusText = acts.length > 0
+ const actStatusText = acts.length > 0
? acts.some((act) => act.outcome === "didnt")
? "Browser action plan did not satisfy its expectations."
: acts.every((act) => act.outcome === "worked")
? "Browser action plan worked."
: "Browser action plan outcome is unknown."
- : waits.length === 0
- ? "Actions executed successfully."
- : failedWait
- ? `Browser condition ${failedWait}.`
- : "Browser condition satisfied.";
+ : undefined;
+ const waitStatusText = waits.length === 0 ? undefined : failedWait ? `Browser condition ${failedWait}.` : "Browser condition satisfied.";
+ const statusText = failedWait ? `Browser condition ${failedWait}.` : actStatusText ?? waitStatusText ?? "Actions executed successfully.";
return { content, details: { statusText, readResults } };
}
@@ -287,6 +285,11 @@
return [`wait_for: ${result.status}/${result.evidence}${reason} after ${result.elapsed_ms}ms`, ...result.details].join("\n");
}
+/**
+ * Formats a browser action plan result for model-facing tool text output.
+ * Includes per-step outcomes and successor details while capping successor
+ * diff lines so large diffs stay in structured `readResults`, not tool text.
+ */
export function formatBrowserActResult(result: BrowserActResult): string {
const lines = [`browser_act: ${result.outcome}`];
if (result.stopped_at !== undefined) lines.push(`stopped_at: ${result.stopped_at} (${result.stop_reason ?? "unknown"})`);You can send follow-ups to the cloud agent here.
f187fc1 to
1ad2322
Compare
abf99f3 to
1bf73c9
Compare
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Baseline wait aborts navigation expects
- I allowed location expectations with an explicit pre-action baseline to continue past initial navigation drift so the poll loop can mark successful navigations as newly verified.
- ✅ Fixed: Mixed batch ignores wait failures
- I updated mixed-batch execution to stop after browser_wait_for results with interrupted, timed_out, or unverifiable statuses, matching the existing browser_act stop behavior.
Or push these changes by commenting:
@cursor push fbc43fc737
Preview (fbc43fc737)
diff --git a/packages/agent/src/translator/browser-wait.ts b/packages/agent/src/translator/browser-wait.ts
--- a/packages/agent/src/translator/browser-wait.ts
+++ b/packages/agent/src/translator/browser-wait.ts
@@ -109,7 +109,9 @@
}
const initial = evaluateBrowserExpectation(options.expect, baseline, baseline, runtime.resolveRef);
if (!baseline.complete) return terminal("unverifiable", "unverifiable", initial, initial, started, now(), "incomplete_observation");
- if (runtime.liveNavigationEpoch(targetId) !== baseline.navigationEpoch || [...baseline.generations].some(([key, generation]) => runtime.liveGeneration(key) !== generation)) return terminal("interrupted", "unverifiable", initial, initial, started, now(), "navigation");
+ const baselineNavigated = runtime.liveNavigationEpoch(targetId) !== baseline.navigationEpoch || [...baseline.generations].some(([key, generation]) => runtime.liveGeneration(key) !== generation);
+ const allowLocationRecovery = !!options.baseline && isLocationExpectation(options.expect);
+ if (baselineNavigated && !allowLocationRecovery) return terminal("interrupted", "unverifiable", initial, initial, started, now(), "navigation");
if (expired()) return timedOut(started, now());
if (initial.reason === "stale_ref") return terminal("interrupted", "unverifiable", initial, initial, started, now(), initial.reason);
if (initial.truth === true && !options.baseline) return terminal("satisfied", "preexisting", initial, initial, started, now());
diff --git a/packages/agent/src/translator/translator.ts b/packages/agent/src/translator/translator.ts
--- a/packages/agent/src/translator/translator.ts
+++ b/packages/agent/src/translator/translator.ts
@@ -136,7 +136,10 @@
await flush();
const reads = await this.browser().execute(action);
result.readResults.push(...reads);
- if (action.type === "browser_act" && reads.some((read) => read.type === "browser_act" && read.result.stop_reason)) break;
+ if (
+ (action.type === "browser_act" && reads.some((read) => read.type === "browser_act" && read.result.stop_reason)) ||
+ (action.type === "browser_wait_for" && reads.some((read) => read.type === "browser_wait_for" && (read.result.status === "interrupted" || read.result.status === "timed_out" || read.result.status === "unverifiable")))
+ ) break;
continue;
}
switch (action.type) {
diff --git a/packages/agent/test/browser-wait.test.ts b/packages/agent/test/browser-wait.test.ts
--- a/packages/agent/test/browser-wait.test.ts
+++ b/packages/agent/test/browser-wait.test.ts
@@ -131,6 +131,23 @@
expect(result).toMatchObject({ status: "satisfied", evidence: "newly_verified" });
});
+ it("allows location expectations to verify when a pre-action baseline is already stale", async () => {
+ let time = 0;
+ const baseline = observation([], true, { url: "https://a.test/start", generations: new Map([["page", 0]]) });
+ const result = await waitForBrowserExpectation({
+ selectTarget: async () => "page",
+ observeTarget: async () => observation([], true, { url: "https://a.test/done", navigationEpoch: 1, generations: new Map([["page", 1]]) }),
+ dialogCount: () => 0,
+ targetExists: async () => true,
+ liveGeneration: () => 1,
+ liveNavigationEpoch: () => 1,
+ resolveRef: missingRef,
+ now: () => time,
+ delay: async (ms) => { time += ms; },
+ }, { expect: { type: "url", changed: true }, baseline, targetId: "page", timeoutMs: 20, pollMs: 10 });
+ expect(result).toMatchObject({ status: "satisfied", evidence: "newly_verified" });
+ });
+
it("reports navigation instead of timeout when a location change lands at the deadline", async () => {
let time = 0, reads = 0;
const result = await waitForBrowserExpectation({
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -300,6 +300,30 @@
await translator.executeBatch([{ type: "browser_act", steps: [{ type: "wait" }] }, { type: "browser_text" }]);
expect(executed).toEqual(["browser_act"]);
});
+
+ it.each(["interrupted", "timed_out", "unverifiable"] as const)("stops a mixed batch after a browser_wait_for %s result", async (status) => {
+ const { client } = createClient();
+ const executed: string[] = [];
+ const result: BrowserWaitForResult = {
+ status,
+ evidence: status === "timed_out" ? "failed" : "unverifiable",
+ initial: { truth: false, details: ["before"] },
+ final: { truth: false, details: ["after"] },
+ elapsed_ms: 1,
+ ...(status === "interrupted" ? { reason: "navigation" } : {}),
+ details: [],
+ };
+ const executor = { execute: async (action: CuaBrowserAction) => {
+ executed.push(action.type);
+ return action.type === "browser_wait_for" ? [{ type: "browser_wait_for", result } as BatchReadResult] : [];
+ } } as unknown as BrowserExecutor;
+ const translator = new InternalComputerTranslator({ browser, client, createBrowserExecutor: () => executor });
+ await translator.executeBatch([
+ { type: "browser_wait_for", expect: { type: "text", text: "Ready" } },
+ { type: "browser_text" },
+ ]);
+ expect(executed).toEqual(["browser_wait_for"]);
+ });
});
describe("InternalComputerTranslator computer additions", () => {You can send follow-ups to the cloud agent here.
1ad2322 to
07370f2
Compare
1bf73c9 to
56c86e1
Compare
|
bugbot run |
56c86e1 to
626c63b
Compare
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Stop reason masks failed expectation
- I reordered stop-reason resolution so failed semantic waits are classified before post-action observation completeness fallbacks.
- ✅ Fixed: Misleading stopped_at on successor race
- I removed the late-boundary fallback that assigned
stopped_atduring successor collection when no step loop interruption occurred.
- I removed the late-boundary fallback that assigned
Or push these changes by commenting:
@cursor push a17aa0b911
Preview (a17aa0b911)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -88,13 +88,14 @@
const postBoundary = after && afterTargets ? boundary(before, after, targets, afterTargets, dialogs, runtime) : undefined;
if (after && afterTargets) { current = after; targets = afterTargets; dialogs = runtime.dialogCount(); }
- stopReason = (!after?.complete ? "control_flow" : undefined)
- ?? postBoundary
+ stopReason = postBoundary
?? waitStopReason(waitResult)
?? (stale ? "stale_ref"
: actionError ? "action_failed"
: expectation?.status === "failed" ? "expectation_failed"
- : expectation?.status === "preexisting" || expectation?.status === "unverifiable" || !after ? "control_flow" : undefined);
+ : undefined)
+ ?? (!after?.complete ? "control_flow" : undefined)
+ ?? (expectation?.status === "preexisting" || expectation?.status === "unverifiable" || !after ? "control_flow" : undefined);
if (stopReason) { stoppedAt = index; break; }
}
@@ -123,7 +124,6 @@
current = observed; targets = successorTargets; dialogs = runtime.dialogCount();
if (lateBoundary) {
stopReason ??= lateBoundary;
- stoppedAt ??= finalStepIndex;
successorError = new Error(`${lateBoundary} changed successor observation`);
continue;
}
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -156,6 +156,13 @@
expect(rt.dispatched).toEqual(["click"]);
});
+ it("keeps expectation_failed when post-action observation fails", async () => {
+ const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Missing" } }] }, runtime([
+ observation("before"), observation("before"), observation("after"),
+ ], [waitResult("failed")], { failObservationAt: 2 }));
+ expect(result).toMatchObject({ outcome: "didnt", stopped_at: 0, stop_reason: "expectation_failed", steps: [{ outcome: "didnt", expectation: { status: "failed" } }] });
+ });
+
it("stops on a stale ref without dispatching later steps", async () => {
const rt = runtime([observation("before"), observation("before"), observation("successor")], [], { dispatchError: new Error("ref e1 is stale") });
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1" }, { type: "type", text: "no" }] }, rt);
@@ -190,6 +197,7 @@
observation("before"), observation("before"), observation("after"), observation("stale"), observation("stable"),
], [waitResult("newly_verified")], { targets: [["target-1"], ["target-1"], ["target-1"], ["target-1", "target-2"], ["target-1", "target-2"]] }));
expect(result).toMatchObject({ stop_reason: "control_flow", successor: { status: "observed", title: "stable", text: "stable" } });
+ expect(result).not.toHaveProperty("stopped_at");
});
it("preserves a timed-out final expectation when the successor satisfies it", async () => {You can send follow-ups to the cloud agent here.
07370f2 to
cf78571
Compare
626c63b to
dc039c6
Compare
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit dc039c6. Configure here.
cf78571 to
eac9a33
Compare
dc039c6 to
6f97b80
Compare
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Final expect upgrade leaves stop reason
- When successor evaluation upgrades an unverifiable final expectation to true, runBrowserAct now clears the inherited stop_reason/stopped_at so the verified result reports worked consistently.
Or push these changes by commenting:
@cursor push 8158041927
Preview (8158041927)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -131,11 +131,16 @@
if (!observed.complete) throw new Error("successor observation incomplete");
if (action.expect && finalExpectation && finalExpectation.status !== "failed") {
const evaluation = runtime.evaluate(action.expect, observed, baseline);
+ const priorFinalStatus = finalExpectation.status;
finalExpectation = {
status: evaluation.truth === true ? finalExpectation.before === true ? "preexisting" : "newly_verified" : evaluation.truth === false ? "failed" : "unverifiable",
before: finalExpectation.before, after: evaluation.truth,
details: [...finalExpectation.details, ...evaluation.details.map((detail) => `successor: ${detail}`)],
};
+ if (evaluation.truth === true && priorFinalStatus === "unverifiable") {
+ stopReason = undefined;
+ stoppedAt = undefined;
+ }
if (evaluation.truth !== true) {
stopReason = evaluation.truth === false ? "expectation_failed" : "control_flow";
stoppedAt = finalStepIndex;
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -199,6 +199,15 @@
expect(result).toMatchObject({ outcome: "didnt", stop_reason: "expectation_failed", final_expectation: { status: "failed", after: false }, successor: { status: "observed", title: "after" } });
});
+ it("clears an unverifiable final stop reason when successor verification succeeds", async () => {
+ const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "wait" }], expect: { type: "url", contains: "after" } }, runtime([
+ observation("before"), observation("before"), observation("after"), observation("after"),
+ ], [waitResult("unverifiable")]));
+ expect(result).toMatchObject({ outcome: "worked", final_expectation: { status: "newly_verified", after: true }, successor: { status: "observed", title: "after" } });
+ expect(result).not.toHaveProperty("stop_reason");
+ expect(result).not.toHaveProperty("stopped_at");
+ });
+
it("does not report incomplete successor observations as authoritative", async () => {
const incomplete = { ...observation("after"), complete: false };
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Done" } }] }, runtime([You can send follow-ups to the cloud agent here.
eac9a33 to
8fd179f
Compare
6f97b80 to
9c821b9
Compare
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Plan expect skipped on navigation
- Terminal navigation now still runs and revalidates the plan-level expectation at the last step even when the step has no per-step expect, and verified plan expectations now count as verified navigation outcomes.
Or push these changes by commenting:
@cursor push f044da8cbe
Preview (f044da8cbe)
diff --git a/packages/agent/src/translator/browser-act.ts b/packages/agent/src/translator/browser-act.ts
--- a/packages/agent/src/translator/browser-act.ts
+++ b/packages/agent/src/translator/browser-act.ts
@@ -101,7 +101,7 @@
}
let finalExpectation: BrowserActExpectationEvidence | undefined;
- const terminalNavigation = stopReason === "navigation" && stoppedAt === finalStepIndex && steps.at(-1)?.outcome === "worked";
+ const terminalNavigation = stopReason === "navigation" && stoppedAt === finalStepIndex;
if (action.expect && (!stopReason || terminalNavigation)) {
try {
const result = await runtime.wait(action.expect!, baseline, baseline.targetId, action.tab_id, action.timeout_ms, action.poll_ms);
@@ -163,7 +163,8 @@
const verified = action.expect
? finalExpectation?.status === "newly_verified"
: steps.length === action.steps.length && steps.every((step) => step.outcome === "worked");
- const verifiedNavigation = stopReason === "navigation" && steps.length === action.steps.length && steps.at(-1)?.outcome === "worked";
+ const verifiedNavigation = stopReason === "navigation" && steps.length === action.steps.length
+ && (steps.at(-1)?.outcome === "worked" || finalExpectation?.status === "newly_verified");
return {
outcome: failed ? "didnt" : !timedOut && verified && (!stopReason || verifiedNavigation) ? "worked" : "unknown",
steps,
diff --git a/packages/agent/test/translator-browser.test.ts b/packages/agent/test/translator-browser.test.ts
--- a/packages/agent/test/translator-browser.test.ts
+++ b/packages/agent/test/translator-browser.test.ts
@@ -185,6 +185,17 @@
expect(result).toMatchObject({ outcome: "worked", stopped_at: 0, stop_reason: "navigation", final_expectation: { status: "newly_verified" } });
});
+ it("evaluates plan expectations after terminal navigation without step expectations", async () => {
+ const before = observation("before");
+ const after = observation("after", 1);
+ const result = await runBrowserAct({
+ type: "browser_act",
+ steps: [{ type: "click", ref: "e1" }],
+ expect: { type: "url", contains: "after" },
+ }, runtime([before, before, after, after], [waitResult("newly_verified")]));
+ expect(result).toMatchObject({ outcome: "worked", stopped_at: 0, stop_reason: "navigation", final_expectation: { status: "newly_verified" } });
+ });
+
it("recollects a successor after a raced target boundary", async () => {
const result = await runBrowserAct({ type: "browser_act", steps: [{ type: "click", ref: "e1", expect: { type: "text", text: "Done" } }] }, runtime([
observation("before"), observation("before"), observation("after"), observation("stale"), observation("stable"),You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 9c821b9. Configure here.
| finalExpectation = { status: "unverifiable", details: [message(error)] }; | ||
| stopReason = "control_flow"; stoppedAt = finalStepIndex; | ||
| } | ||
| } |
There was a problem hiding this comment.
Plan expect skipped on navigation
Medium Severity
When a plan stops on the last step with stop_reason navigation but that step has no per-step expect (so its outcome stays unknown), the plan-level expect wait is skipped entirely. The successor path also only re-checks plan expect when a final wait already ran, so the declared plan expectation is never evaluated even though a stable successor observation is returned.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9c821b9. Configure here.
8fd179f to
7b76f06
Compare
9c821b9 to
8056413
Compare



summary
timeout_msindependently to each semantic expectation while allowing dispatched input and in-flight observations to settlestack
base:
hypeship/browser-semantic-waitssize
912 changed lines: 877 additions, 35 deletions.
testing
npm run typechecknpm test --workspace @onkernel/cua-ai— 173 passedgit diff --check hypeship/browser-semantic-waits...HEADThe unchanged
ptywrightcompiled tests still fail under Node 22 because extensionless ESM imports cannot resolve fromdist; this stack does not touch that package.Note
Medium Risk
Touches core browser automation (CDP actions, waits, batch control flow) with substantial new orchestration logic; mitigated by broad tests but behavior at navigation/dialog boundaries is easy to get wrong in production.
Overview
Adds
browser_act, a ref/focus-only dependent step list (up to 20 steps) with optional per-step and plan-level semantic expectations, wired through the existing wait engine using a supplied baseline so postconditions are verified without treating already-true conditions as success.runBrowserActruns steps in order, stops on failed expectations, stale refs, navigation/dialog/popup boundaries, and incomplete observations, then returnsworked/didnt/unknown, stop reasons, and a successor snapshot plus a normalized AX diff (refs stripped for stability). Batch tooling formats that result (diff capped at 200 lines) and picks status text from act outcomes; mixed batches stop after a planstop_reasonor a non-satisfiedbrowser_wait_for.Supporting tweaks:
diffObservations, wait navigation-epoch stability checks,renderObservation(..., comparePrevious)for successor text,num_clicks1–3 validation on clicks, and theacttool in browser/hybrid schemas.Reviewed by Cursor Bugbot for commit 9c821b9. Bugbot is set up for automated code reviews on this repo. Configure here.