[STG-2503] fix(evals): generate core browse CLI contract from oclif manifest, fix stale commands

[shrey150]

Summary

Follow-up to Miguel's review comment on #2299: #2299 (comment)

the changes here don't impact the bench/ runners (claude-code/codex). This is only for the core benchmarks that compare tool vs tool, deterministically. The contract here is quite stale now, we should address the changes in a follow up PR.

#2299 fixed the browse CLI v0.9.1 contract for the bench path (framework/claudeCodeToolAdapter.ts, shared by the claude_code/codex external harnesses). This PR does the deferred core half: core/tools/browse_cli.ts, the browse_cli tool-surface implementation used only by the deterministic core tool-vs-tool comparison suite (core/contracts/tool.ts's ToolSurface union: browse_cli, playwright_code, understudy_code, cdp_code, playwright_mcp, chrome_devtools_mcp).

The actual drift (verified live against a local packages/cli build)

core/tools/browse_cli.ts hand-encoded the CLI's command/flag surface independently of framework/claudeCodeToolAdapter.ts, with no link back to the CLI's real command definitions. Both files drift independently; this file's copy was far more stale than Miguel's 4 illustrative examples suggested — full list found by auditing every command this file invokes against packages/cli/src/commands/**:

Method	Called (stale)	Correct now	Failure mode
start()	env local/env remote	(removed — mode flag threaded through instead)	command not found
newPage()	newpage [url]	tab new [url]	command not found
fetchPages()	pages	tab list (returns {tabs: [...]}, not {pages: [...]})	command not found
selectPage()	tab_switch <i>	tab switch <i>	command not found
closePage()	tab_close <i>	tab close <i>	command not found
click(x,y)	click_xy <x> <y>	mouse click <x> <y>	command not found
hover(x,y)	hover <x> <y>	mouse hover <x> <y>	command not found
scroll()	scroll <x> <y> <dx> <dy>	mouse scroll <x> <y> <dx> <dy>	command not found
goto()	open <url> -t <ms>	open <url> --timeout <ms>	Nonexistent flag
wait() (selector)	wait selector <s> -t <ms> -s <state>	wait selector <s> --timeout <ms> --state <state>	Nonexistent flag
wait() (load)	wait load <state> -t <ms>	wait load <state> --timeout <ms>	Nonexistent flag
type() (selector)	fill <s> <text> --no-press-enter	fill <s> <text> (Enter-unpressed is the default)	Nonexistent flag
screenshot()	screenshot -f -t <type> -q <quality>	screenshot --full-page --type <type> --quality <quality> --base64	Nonexistent flag, and without --base64 the CLI now writes a file to disk and returns its path instead of image bytes — a silent behavior change, not just a flag rename

Also fixed: tab.new's old return type declared a created field that was never real (Miguel's note) — dropped it.

The fix: generate the contract instead of hand-typing it twice

packages/cli is built with oclif, which already emits packages/cli/oclif.manifest.json at build time — a JSON manifest of every command and its flags, generated straight from each command class's own static flags. packages/evals/scripts/generateBrowseCliContract.ts reads that manifest into a committed, typed module (core/tools/browseCliContract.generated.ts) exporting BROWSE_CLI_CONTRACT, commandAcceptsFlag(), and BROWSE_CLI_MODELESS_COMMANDS.

core/tools/browse_cli.ts now derives its "which commands reject --local/--remote" list from this generated contract instead of a hand-typed Set(["stop","status"]) — which incidentally surfaced a second latent bug: cdp and daemon also lack --local/--remote in the real CLI and were never in the old hand-typed list either (harmless today since this file never calls them, but a generated list is now correct by construction instead of by luck).

A staleness-guard test (tests/core/browseCliContract.test.ts) rebuilds the contract from a fresh manifest and deep-equals it against the committed file, so a packages/cli change that isn't followed by regenerating fails CI immediately instead of surfacing as a runtime "Nonexistent flag" days later.

Not in scope: framework/claudeCodeToolAdapter.ts's equivalent hand-typed stop/status check only exists on the still-unmerged #2299 (this branch is off origin/main, which doesn't have #2299's fix yet). Generate-ifying it here would either redo #2299's fix or edit code that doesn't exist on main yet. Swapping it to BROWSE_CLI_MODELESS_COMMANDS is a one-line follow-up once #2299 merges.

No changeset: @browserbasehq/stagehand-evals is private: true, never published.

E2E Test Matrix

Command / flow	Observed output	Confidence / sufficiency
pnpm --dir packages/cli build then pnpm --dir packages/evals generate:browse-cli-contract	Manifest built at packages/cli/oclif.manifest.json; generator wrote browseCliContract.generated.ts	Proves the generator runs against a real build and the committed file matches a fresh one (also covered by the staleness test below in CI).
pnpm --dir packages/evals lint (prettier + eslint + typecheck)	All clean, 0 errors, 0 warnings	Full static verification of the diff.
pnpm --dir packages/evals test:unit (vitest run)	357 passed across 49 test files, 0 failed, including the new staleness-guard test	Full unit suite green, contract-vs-manifest deep-equal passes.
Negative controls — every stale command/flag invoked directly against <local build> of packages/cli (click_xy, newpage, hover, scroll, tab_switch, tab_close, env, pages, open ... -t, wait ... -t -s, screenshot -f, fill ... --no-press-enter)	Every one failed: "<cmd>" is not a browse command or Error: Nonexistent flag	Proves the drift is real, not a false alarm — this is the current, live, broken state of main.
Real E2E — BrowseCliTool driven directly (not mocked) against <local build> of packages/cli in LOCAL mode: newPage, wait (all 3 kinds), title/url (get), listPages/selectPage/closePage (tab list/switch/close), click/hover/scroll at coords (mouse click/hover/scroll), click/press by selector, type via fill (no flag), screenshot with --base64, represent (snapshot)	All succeeded against https://example.com: real tab targetIds, title: "Example Domain", valid PNG bytes (15988 bytes, correct magic number) from screenshot, snapshot content returned, session cleaned up via stop --force with no orphaned daemon process afterward	Proves every corrected command works end-to-end against the real driver, not just that it typechecks — this is the strongest evidence in this table.
Compatibility — this branch is off origin/main and touches only core/tools/browse_cli.ts + new files; framework/claudeCodeToolAdapter.ts untouched	git diff --stat confirms 0 lines changed in that file	Proves this PR and #2299 cannot conflict — safe to merge in either order.

Linear

STG-2503 — https://linear.app/browserbase/issue/STG-2503/fixevals-generate-core-browse-cli-contract-from-oclif-manifest-fix

---

Summary by cubic

Generates the core browse_cli contract from the oclif manifest and updates core/tools/browse_cli.ts to match the current CLI, fixing stale commands/flags used by core evals. Addresses STG-2503 with a CI guard and a unit test for mode flag injection.

• Bug Fixes

  • Mode flags: per-command --local/--remote (removed env).
  • Command names: tab new/list/switch/close, mouse click/hover/scroll (replaced newpage, pages, tab_switch, tab_close, click_xy, hover, scroll).
  • Flags: --timeout/--state; removed --no-press-enter.
  • screenshot returns bytes via --base64 (supports --full-page/--type/--quality).
  • Return shapes: tab list → {tabs: [...]}; dropped nonexistent created.

• Refactors

  • Added generator packages/evals/scripts/generateBrowseCliContract.ts to read packages/cli/oclif.manifest.json and emit browseCliContract.generated.ts; added pnpm generate:browse-cli-contract.
  • browse_cli.ts derives modeless commands from the contract and appends mode flags only when accepted.
  • Added staleness test to fail CI on contract drift and a unit test for mode-flag behavior.
  • Generator hardening: use isMainModule() and normalize displayed paths.
  • Removed an outdated historical comment in the core browse_cli adapter.

^{Written for commit 6068ab3. Summary will update on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 6068ab3

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

3 issues found across 5 files

Confidence score: 3/5

• In packages/evals/scripts/generateBrowseCliContract.ts, the import.meta.url entrypoint check is not Windows-safe, so main() may never run there and contract generation can silently fail for Windows contributors/CI — switch to the existing runtimePaths.ts helper for a cross-platform invocation check before merging.
• In packages/evals/core/tools/browse_cli.ts, mode-flag injection now affects all browse_cli invocations, but there’s no focused argv test for modeful vs. modeless command behavior; this leaves a user-facing regression path unguarded — add a targeted unit test around argv construction to de-risk merge.
• In packages/evals/scripts/generateBrowseCliContract.ts, Windows backslashes in generated repo-relative paths can create inconsistent logs/errors and copy-paste friction against the repo’s slash convention — normalize output with template paths or path.posix.join before merging.

Architecture diagram

sequenceDiagram
    participant Dev as Developer
    participant CLI as Browse CLI (oclif build)
    participant Manifest as oclif.manifest.json
    participant Gen as generateBrowseCliContract.ts
    participant Contract as browseCliContract.generated.ts
    participant BrowseCli as browse_cli.ts (BrowseCliRuntime)
    participant EvalTest as Core Evaluation Test
    participant GuardTest as Staleness Guard Test

    Note over Dev,GuardTest: PR: Generate CLI Contract from oclif Manifest

    Dev->>CLI: pnpm --dir packages/cli build
    CLI->>CLI: Build commands from src/commands/**
    CLI->>Manifest: Write oclif.manifest.json (JSON manifest of commands/flags)

    Dev->>Gen: pnpm --dir packages/evals generate:browse-cli-contract
    Gen->>Manifest: Read manifest JSON
    Manifest-->>Gen: Raw command/flag data
    Gen->>Gen: Parse & transform to BrowseCliContract (commandId → {flags: [...]})
    Gen->>Contract: Write browseCliContract.generated.ts (typed module)

    Note over Contract: Exports: BROWSE_CLI_CONTRACT, BrowseCliCommandId,<br/>commandAcceptsFlag(), BROWSE_CLI_MODELESS_COMMANDS

    EvalTest->>BrowseCli: Start session (ToolStartResult)
    BrowseCli->>BrowseCli: Construct BrowseCliRuntime(session, modeFlag)
    alt Modeless command (e.g., stop, status, cdp, daemon)
        BrowseCli->>Contract: BROWSE_CLI_MODELESS_COMMANDS.includes(commandId)
        Contract-->>BrowseCli: true
        BrowseCli->>CLI: execFile(entrypoint, [argv, --json, --session, ...])
    else Commands accepting --local/--remote
        BrowseCli->>Contract: BROWSE_CLI_MODELESS_COMMANDS.includes(commandId)
        Contract-->>BrowseCli: false
        BrowseCli->>CLI: execFile(entrypoint, [argv, --json, --local, --session, ...])
    end

    CLI-->>BrowseCli: stdout: JSON (tabs: [...], base64 string, etc.)

    BrowseCli->>CLI: Command: tab:new [url]
    CLI-->>BrowseCli: {targetId, url, index} (no 'created' field)

    BrowseCli->>CLI: Command: tab:list
    CLI-->>BrowseCli: {tabs: [{index, url, targetId}]}

    BrowseCli->>CLI: Command: mouse:click x y
    BrowseCli->>CLI: Command: mouse:hover x y
    BrowseCli->>CLI: Command: mouse:scroll x y dx dy

    BrowseCli->>CLI: Command: open url --timeout <ms>
    BrowseCli->>CLI: Command: wait selector <s> --timeout <ms> --state <state>
    BrowseCli->>CLI: Command: wait load <state> --timeout <ms>

    BrowseCli->>CLI: Command: fill <s> <text> (no --no-press-enter flag)

    BrowseCli->>CLI: Command: screenshot --base64 --full-page --type <type> --quality <quality>
    CLI-->>BrowseCli: {base64: "..."}
    BrowseCli->>BrowseCli: Buffer.from(base64, "base64")

    Note over GuardTest: Staleness Guard (CI)
    GuardTest->>Manifest: Read fresh manifest
    Manifest-->>GuardTest: Raw JSON
    GuardTest->>GuardTest: buildBrowseCliContractFromManifest()
    GuardTest->>Contract: deep-equal fresh contract vs committed BROWSE_CLI_CONTRACT
    alt Contracts match
        GuardTest-->>GuardTest: Test passes
    else Contracts differ
        GuardTest-->>GuardTest: Test fails → CI fails → regenerate required
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>