[STG-2473] feat(cli): surface navigation httpStatus on browse open

[shrey150]

TL;DR

browse open (and reload/back/forward) now includes the HTTP status of the top-level document on its result — one optional field, httpStatus. That's the whole change. A normal 200 page is unchanged; a navigation with no captured response omits the field.

Why it matters: today, when you open a URL that returns a 404/403, the browser still "loads" something (often a rendered error page or chrome-error://), and an agent driving the CLI can't tell that apart from a real page — so it happily continues as if the nav succeeded. Surfacing the status gives the driver a fact it can act on.

Heads up for the reviewer: I'm opening this as a draft on purpose. I ran a fairly serious A/B (details below) and the honest result is that this is a correct, harmless primitive that doesn't move the aggregate eval number at Haiku. I have a specific ask for you at the bottom — mainly should I keep pursuing this, and should I re-run the eval on Sonnet instead of Haiku?

Linear: STG-2473

---

Where this came from (a customer investigation)

An internal customer evaluation showed self-hosted Sonnet 4.5 + browse CLI underperforming our managed Agents product on multi-step travel/booking tasks. Digging in, the gap wasn't the model — it was scaffolding. Managed agent-runner re-observes and recovers after a bad step; a thin harness driving raw browse CLI didn't even know a step was bad, because a loaded 404 looks like success.

So the question this PR set out to answer: can we make browse "self-documenting" enough that a thin harness recovers like agent-runner does — and does that actually help, measured honestly?

---

Approaches considered

I prototyped three layers of "self-documentation" and tried to isolate each one:

Approach	What it added to the CLI output	Verdict	Why
Candidate links	On a 4xx, scrape the page's own same-domain links and list them so the agent can re-route	❌ Cut	Overfit to booking/search sites; it's really the job of browse snapshot, not nav. No clean signal in evals.
Hint string	On a 4xx, add a one-line hint: "Navigation returned 404 — the page is unavailable; check the URL…"	❌ Not in this PR	Statistically real on the tasks it targets, but it over-abandons recoverable tasks (see data). Net wash, and it's guidance baked into output, which we'd rather keep in SKILL.md.
httpStatus fact	Just the numeric status on the nav result	✅ This PR	It's a plain, correct fact (the actual HTTP status). No regressions, mild positive on the target tasks, and it lets the downstream harness/model decide what to do rather than us prescribing it.

The principle I landed on: facts belong in tool output, guidance belongs in SKILL.md / the harness prompt. httpStatus is a fact. The hint was guidance-in-output, so it's out.

---

The eval (this is the part I want your eyes on)

I built a proper A/B instead of eyeballing trajectories. Setup:

• Harness: the claude_code eval path driving the real browse CLI, on Browserbase remote.
• Scoring: ground-truth V3Evaluator rubric (not the agent's self-report). Judge model = gemini-2.5-flash.
• Model under test: claude-haiku-4-5.
• max_turns = 60 — this matters (see "validity" below).
• 12 tasks × 8 trials × 3 arms = 288 runs. Tasks are block-/4xx-prone travel + booking (incl. 4 real-world flight-booking tasks from the customer eval) plus a couple of easy controls.
• 3 arms: A = baseline (stock CLI) · B = httpStatus only · C = httpStatus + hint.

Results

Metric	A: baseline	B: httpStatus	C: httpStatus + hint
Overall pass	25% (24/96)	27% (26/96)	25% (24/96)
Capped (hit turn limit)	46%	45%	34%
Floor tasks pass (capeair/united/expedia/kayak, n=32)	0%	9%	16%

Significance (two-proportion z):

• Overall: no effect. z(A→B) = +0.33 (p≈0.74); z(A→C) = 0.00. Flat.
• Floor tasks (the 4xx tasks it's designed for): real. z(A→B) = +1.77; z(A→C) = +2.33 (p≈0.02). The hint roughly doubles the fact's effect on those tasks.

How to read this honestly

1. Nothing moves the aggregate. All three arms are ~25%. If the bar is "improves overall task success at Haiku," neither the fact nor the hint clears it.
2. The mechanism is real on the target subgroup. On the dead-URL/4xx tasks, we go 0% → 9% (fact) → 16% (fact+hint). That's the thing working as designed — just diluted to nothing once you average in tasks where HTTP status is irrelevant.
3. The hint is a double-edged blade — this is the most interesting finding. It cuts the cap rate 46% → 34% (agent stops flailing on dead-ends) and lifts the floor tasks the most — but it also makes Haiku give up on recoverable tasks: santorini_flight went 3/8 → 0/8 (capped 2 → 0: it bailed instantly), and ticketmaster_sporting went 4/8 → 2/8 (the fact-only arm had lifted it to 7/8). The over-abandonment cancels the floor gains → net wash. The hint text is too blunt for a small model to apply judiciously.
4. httpStatus-only has no regressions and is a correct primitive, which is why it's the safe thing to land.

A validity note (why I trust this run and not my earlier ones)

An earlier factorial ran at max_turns = 30 and showed no signal anywhere. That was an artifact: 72% of runs hit the 30-turn cap, and a capped run passes ~2% — so it was measuring turn-budget, not the change. At max_turns = 60 the cap rate drops to ~46% and, crucially, becomes correlated with genuine task difficulty (easy tasks cap 0%, kayak caps 100% in both arms because it's just turn-starved) rather than a uniform guillotine. That's why this run's deltas are meaningful and the old one's weren't.

---

What I actually want advice on

1. Is this worth pursuing at all? httpStatus is harmless and correct, but the eval doesn't prove it lifts task success at Haiku. Is "correct primitive with a real-but-small subgroup effect" a good enough reason to land, or should I hold it?
2. Haiku vs Sonnet. The over-abandonment regressions smell like a small-model artifact — a stronger model would likely use the status more judiciously. Our real deployment (agent-runner) is Sonnet-class. I think the haiku result is a floor on the value, not a ceiling. Should I re-run the A/B (at least the 4 floor tasks) on Sonnet before we decide? Cheap — ~30 min.
3. The hint. Given the fact-vs-guidance principle, if we want the hint's floor-task lift, should it live as guidance in SKILL.md / the harness prompt (persistence-preserving wording) rather than in tool output? Worth a separate experiment?

---

Scope of this PR

Deliberately tiny — just the fact:

• openResult() sets httpStatus when a navigation response was captured (session-manager.ts).
• Nav handlers read response.status() from open/reload/back/forward (navigation.ts).
• OpenResult.httpStatus?: number type (types.ts).
• SKILL.md: one factual line documenting the field (no prescriptive "give up" guidance).
• Unit tests + changeset (browse patch).

No candidate links, no hint string — both cut per the data above.

---

E2E Test Matrix

Command / flow	Observed output	Confidence / sufficiency
browse open https://httpbin.org/status/404 --remote (local build)	"httpStatus": 404, url: "chrome-error://chromewebdata/", no hint field	Proves the fact is surfaced on a real 4xx and the hint is gone. This is the exact dead-end an agent would otherwise read as success.
browse open https://example.com --remote (local build)	"httpStatus": 200, no hint field	Proves a normal page carries the status and is otherwise unchanged.
vitest run tests/self-documenting.test.ts	3/3 pass (404→status, 200→status, no-response→omitted)	Unit-level guarantee of the openResult contract.
pnpm build (packages/cli)	tsc + oclif manifest clean	Typecheck/build green.
Powered A/B — 288 runs, Haiku, max_turns=60, V3Evaluator-scored, on Browserbase	baseline 25% · httpStatus 27% · +hint 25%; floor tasks 0%→9%→16%	The real evidence. Shows httpStatus is harmless overall and positive-but-small on the 4xx subgroup; motivates the Sonnet question above. Does not prove an aggregate win at Haiku.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 02afd18

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[shrey150]

instead of this hint, should we add it into skill.md? does this hint meaningfully improve agent task performance, or can we forgo it assuming that agents can figure out a 4xx / 5xx error is bad