feat(core): add setInputFiles to act() and observe()

[AntonVishal]

Fixes #972

Thanks @shrey150 for the base #1718 for the snapshot/a11y groundwork there.

It's worked in a real job application flow - https://www.browserbase.com/sessions/a18ed533-de39-4684-b2bf-eaf8faeb04d3

This PR takes the next step setInputFiles as a first-class AI action so you can upload files through act() without manually unpacking an observe result and calling page.locator().setInputFiles().

why

observe() on file inputs was returning click on a wrapper div instead of the actual <input type="file"> (see #972). Even after the observe context fix in #1718, there was still no way to say "upload this file" in one act() call — you had to wire up the selector yourself.

On Browserbase, remote act() also can't read files off your laptop, so uploads need a local execution path even when observe runs server-side.

what changed

• add setInputFiles to the supported action methods in inference + prompts (act + observe)
• thread input[type] metadata through the snapshot pipeline; keep file inputs during a11y pruning and synthesize them when AX drops them (carried over from Fix observe context for file inputs; add upload eval/example #1718)
• new fileUploadActUtils.ts: detect upload instructions with file-path variables, resolve %key% placeholders, disambiguate when multiple file inputs match
• on Browserbase: remote observe() to find the input, local setInputFiles to push the file from the developer machine
• update observe_file_uploads bench eval to assert method === "setInputFiles"
• add act_file_upload_variables bench eval — ran it on Browserbase, session 360fc05f-30b1-4a2c-a1f5-e0597bbfd1cb
• docs: file upload section in act.mdx
• changeset: minor bump for @browserbasehq/stagehand

Usage:

await stagehand.act("upload %resume% to the resume field", {
  variables: { resume: "/path/to/resume.pdf" },
});

test plan

• unit: file-upload-act-utils.test.ts, v3-file-upload-act.test.ts, snapshot/a11y tests, prompt tests
• integration: setinputfiles.spec.ts (locator, deepLocator iframe, act with variables, payload objects)
• bench eval act_file_upload_variables — green on Browserbase (session)
• pnpm --filter @browserbasehq/stagehand test (unit)
• pnpm --filter @browserbasehq/stagehand test:integration (if you want the browser suite)

---

Summary by cubic

Adds file upload support to act() and observe() via the new setInputFiles action with smarter variable resolution, input disambiguation, and local execution when using the hosted API. This improves reliability and makes uploads a first-class action in @browserbasehq/stagehand.

• New Features

  • Support setInputFiles in act()/observe(); accepts one or more file paths or %var% placeholders.
  • Resolves variables and infers file paths when arguments are empty; understands %tokens% and whole-word variable names; supports existing extensionless paths; clear errors when multiple inputs tie.
  • Hosted flow: detects upload intent, bypasses act-cache, runs observe() remotely and performs setInputFiles locally; also executes locally when an Action with method: "setInputFiles" is passed.
  • Prompts updated to choose setInputFiles for “input, file” (never click) and pass file paths/placeholders; schema descriptions updated; docs updated with an uploads section; new bench eval act_file_upload_variables.

• Bug Fixes

  • Snapshot/a11y now keeps and synthesizes <input type="file"> (including within scoped trees) so observe() targets the real input instead of wrappers.
  • DOM snapshot threads input[type] and XPath metadata and respects scoping/ignored nodes, improving file input detection and upload targeting.

^{Written for commit ef2bc4d. Summary will update on new commits.}

[changeset-bot]

🦋 Changeset detected

Latest commit: ef2bc4d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Minor
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run.
Approving the latest commit mirrors it into an internal PR owned by the approver.
If new commits are pushed later, the internal PR stays open but is marked stale until someone approves the latest external commit and refreshes it.

[cubic-dev-ai]

6 issues found across 25 files

Confidence score: 3/5

• The highest-risk area is packages/core/lib/prompt.ts (act()) together with packages/core/lib/v3/handlers/handlerUtils/fileUploadActUtils.ts: lookup-style phrasing can be misrouted into setInputFiles, while non-upload|attach wording (e.g., select/choose) can miss the intended local upload path, creating inconsistent and potentially wrong file actions for users — tighten the act/step-two prompt gating and broaden intent matching before merging.
• In packages/core/lib/v3/understudy/a11y/snapshot/a11yTree.ts, scoped observe/extract can fall back to a full-page snapshot when a selector targets a file input omitted from the AX tree, which can cause the model to act on the wrong DOM context — persist/use the resolved backend id even without an AX-node match so scoped targeting is preserved.
• packages/core/lib/v3/handlers/handlerUtils/fileUploadActUtils.ts looksLikeFilePath() is overly broad (any dotted token), so normal values like emails/domains/versions may be treated as file paths and push act() down invalid upload handling — require stricter path signals (extension/path separators or existence checks) before routing as upload.
• packages/core/tests/unit/timeout-handlers.test.ts currently allows false positives by not asserting the returned result in two cases, so regressions (success: false) could ship undetected; packages/evals/tasks/bench/act/act_file_upload_variables.ts also has minor temp-file cleanup leakage on teardown failure — add explicit result assertions and move fixture cleanup into a finally block to de-risk merge.

Architecture diagram

sequenceDiagram
    participant User
    participant Stagehand as Stagehand V3
    participant ActCache as act cache
    participant RemoteAPI as Remote observe/act
    participant Snapshot as Snapshot/A11y pipeline
    participant FileUtils as fileUploadActUtils
    participant ActHandler as ActHandler
    participant Locator as page.locator

    User->>Stagehand: act("upload %resume% to the resume field", {variables: {resume: "/tmp/resume.pdf"}})

    Stagehand->>Stagehand: shouldResolveFileUploadLocally(instruction, variables)
    Note over Stagehand: detects upload verb + file path variables<br/>(NEW: fileUploadActUtils)

    alt Hosted + file upload (isFileUploadAct = true)
        Stagehand->>ActCache: skip (bypass cache for uploads)
        Stagehand->>RemoteAPI: observe(instruction)
        RemoteAPI->>Snapshot: getTree(focusFrame)
        Note over Snapshot: NEW: preserve file inputs in a11y tree<br/>(inputTypeMap, appendMissingFileInputNodes)
        Snapshot-->>RemoteAPI: accessibility tree with setInputFiles actions
        RemoteAPI-->>Stagehand: actions[] (including setInputFiles)

        Stagehand->>FileUtils: selectFileUploadAction(actions, instruction, variables)
        Note over FileUtils: disambiguates multiple file inputs<br/>(throws if ambiguous)
        FileUtils-->>Stagehand: selected action (e.g. "xpath=//input#resume")

        Stagehand->>ActHandler: takeDeterministicAction(action, page, timeout, llmClient, variables, instruction)

        ActHandler->>FileUtils: resolveSetInputFilesArguments(action, variables, instruction)
        Note over FileUtils: substitutes %resume% → "/tmp/resume.pdf"<br/>or infers from description/instruction
        FileUtils-->>ActHandler: resolved file paths ["/tmp/resume.pdf"]

        ActHandler->>Locator: setInputFiles(["resume.pdf"])
        Locator-->>ActHandler: success

        ActHandler-->>Stagehand: ActResult {success: true, method: "setInputFiles"}

    else Local execution or non-upload instruction
        Stagehand->>ActCache: prepareContext (if enabled)
        Stagehand->>RemoteAPI: act(input, options) — normal path
        Note over RemoteAPI: LLM inference uses updated prompt<br/>(NEW: prefers setInputFiles for file inputs)
        RemoteAPI-->>Stagehand: ActResult
    end

    Stagehand-->>User: ActResult

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 7 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>