fix(verifier): configurable canonical-evidence cap + required-field precedence rule

[miguelg719]

Why

Two verifier false-verdict sources found while analyzing eval runs:

1. collectCanonicalEvidence cut every text blob at a fixed 4k chars. Full a11y trees are routinely 30–85k chars and the datum a criterion needs often sits deep in the tree — the cut silently dropped it downstream, producing false negatives on large-page tasks.
2. The judge inferred field required-ness from the aria tree's required attribute, which shared form components duplicate onto optional fields (e.g. "First Name (Legal)" carrying the marker of the truly-required "First Name (Chosen)") — also false negatives on "all required fields filled" criteria.

What changed

• canonicalEvidenceChars joins VerifierConfig's truncation section — resolved in resolveVerifierConfig via the existing readChars helper (VERIFIER_CANONICAL_EVIDENCE_CHARS, default 4000, honors any positive value) and lifted by the truncation master switch, env or programmatic truncation.disabled — matching every other verifier knob. The cap is threaded through collectCanonicalEvidence's options; no inline env reads.
• Judgment rule 9 (precedence, not a ban): when screenshots show a form field, the visual required marker takes precedence over a conflicting aria attribute; aria-only evidence remains a (weak) signal rather than being ignored — so forms whose only required indicator is the attribute still grade correctly.

Note: with the master switch on, canonical blobs are unbounded like every other lifted limit — that's the documented contract for high-context judge models; operators opt in.

Tests

• collectCanonicalEvidence: default 4k cap, sub-default override (100), lifted cap (9k survives) — option-based, no env mutation.
• resolveVerifierConfig: default, env value below the old floor (2000 honored), master switch via env and via programmatic override.
• typecheck + build green; verifier evidence/rubric suites 10/10.

🤖 Generated with Claude Code

---

Summary by cubic

Make the canonical-evidence text cap configurable and liftable to avoid dropping deep aria data on large pages. Update required-field judgment to prefer visual required markers over conflicting aria attributes.

• Bug Fixes

  • Add truncation.canonicalEvidenceChars to VerifierConfig; reads VERIFIER_CANONICAL_EVIDENCE_CHARS (default 4000) and is lifted by VERIFIER_DISABLE_TRUNCATION or programmatic truncation.disabled. Threaded through collectCanonicalEvidence so each text blob is capped consistently.
  • When screenshots show a form field, prefer the visual required marker (e.g., asterisk, “required” text) over the aria required attribute on conflict; aria-only remains a weak signal.

• Migration

  • Optional: set VERIFIER_CANONICAL_EVIDENCE_CHARS to tune the per-blob cap, or VERIFIER_DISABLE_TRUNCATION=1 to lift all truncation. No changes needed for default behavior.

^{Written for commit ebcef8f. Summary will update on new commits.}

[changeset-bot]

🦋 Changeset detected

Latest commit: ebcef8f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@browserbasehq/stagehand	Patch
@browserbasehq/stagehand-evals	Patch
@browserbasehq/stagehand-server-v3	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[cubic-dev-ai]

2 issues found across 7 files

Confidence score: 3/5

• In packages/core/lib/v3/verifier/evidence.ts, accepting non-positive or invalid canonicalEvidenceChars can cause canonical evidence to be truncated incorrectly or dropped, which risks incorrect verifier outputs when overrides are used — validate this as a positive integer and fall back to the default before merging.
• In packages/core/lib/v3/verifier/rubricVerifier.ts, the implicit canonical-evidence cap is duplicated across resolveVerifierConfig and collectCanonicalEvidence, so future edits can drift and create inconsistent behavior across code paths — move to a shared constant/single fallback source to de-risk maintenance regressions.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/core/lib/v3/verifier/evidence.ts">

<violation number="1" location="packages/core/lib/v3/verifier/evidence.ts:305">
P2: Programmatic overrides can silently suppress or corrupt canonical text evidence because `canonicalEvidenceChars` is accepted without validating it is a positive integer. Clamping invalid values to the default (or minimum 1) keeps the new configurability without introducing evidence-loss edge cases.</violation>
</file>

<file name="packages/core/lib/v3/verifier/rubricVerifier.ts">

<violation number="1" location="packages/core/lib/v3/verifier/rubricVerifier.ts:329">
P3: Canonical evidence default is now defined in two places, so a future change can make `resolveVerifierConfig` and `collectCanonicalEvidence` disagree about the implicit cap. A shared constant (or single-source fallback) would keep behavior aligned across code paths.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Admin as Admin/DevOps (env vars)
    participant Consumer as Programmatic caller
    participant Config as resolveVerifierConfig
    participant Evidence as collectCanonicalEvidence
    participant Judge as RubricVerifier.judge
    participant LLM as LLM Judge

    Note over Admin,Consumer: Configuration resolution
    
    alt Environment-based
        Admin-->>Config: VERIFIER_CANONICAL_EVIDENCE_CHARS=2000
        Admin-->>Config: VERIFIER_DISABLE_TRUNCATION=1 (master switch)
    else Programmatic override
        Consumer-->>Config: { truncation: { disabled: true } }
    end
    
    Config->>Config: readChars() helper: parse env or fallback to 4000,<br/>lift to MAX_SAFE_INTEGER if master switch on
    Config-->>Consumer: resolved config with truncation.canonicalEvidenceChars

    Note over Evidence,Judge: Evidence collection & judgment
    
    Consumer->>Evidence: collectCanonicalEvidence(trajectory, { canonicalEvidenceChars })
    Evidence->>Evidence: For each text blob, slice to canonicalEvidenceChars chars
    alt Large a11y tree (e.g., 50k chars)
        Evidence->>Evidence: Truncate to cap (default 4000, or lifted value)
        Note over Evidence: If cap too small, deep data lost → false negative risk
    end
    Evidence-->>Consumer: evidence list (text + images)

    Consumer->>Judge: judge({ evidence, config })
    Judge->>Judge: Build judgment prompt including rule 9
    Note over Judge,LLM: Rule 9: Visual required marker > aria required attribute on conflict<br/>Aria-only fields still treated as weak signal
    Judge->>LLM: Submit evidence + prompt
    LLM-->>Judge: Verdict with per-criterion scores
    Judge-->>Consumer: Verdict

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Programmatic overrides can silently suppress or corrupt canonical text evidence because canonicalEvidenceChars is accepted without validating it is a positive integer. Clamping invalid values to the default (or minimum 1) keeps the new configurability without introducing evidence-loss edge cases.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/verifier/evidence.ts, line 305:

<comment>Programmatic overrides can silently suppress or corrupt canonical text evidence because `canonicalEvidenceChars` is accepted without validating it is a positive integer. Clamping invalid values to the default (or minimum 1) keeps the new configurability without introducing evidence-loss edge cases.</comment>

<file context>
@@ -295,6 +295,14 @@ export async function collectCanonicalEvidence(
+  // production caller resolves this from VerifierConfig's truncation section
+  // (VERIFIER_CANONICAL_EVIDENCE_CHARS, lifted by the truncation master
+  // switch); direct callers get the historical 4k default.
+  const canonicalEvidenceChars = opts.canonicalEvidenceChars ?? 4000;
   const seenText = new Set<string>();
   const addTextEvidence = (
</file context>

Suggested change

	const canonicalEvidenceChars = opts.canonicalEvidenceChars ?? 4000;
	const canonicalEvidenceChars =
	Number.isFinite(opts.canonicalEvidenceChars) &&
	(opts.canonicalEvidenceChars as number) > 0
	? Math.floor(opts.canonicalEvidenceChars as number)
	: 4000;

[cubic-dev-ai]

P3: Canonical evidence default is now defined in two places, so a future change can make resolveVerifierConfig and collectCanonicalEvidence disagree about the implicit cap. A shared constant (or single-source fallback) would keep behavior aligned across code paths.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/verifier/rubricVerifier.ts, line 329:

<comment>Canonical evidence default is now defined in two places, so a future change can make `resolveVerifierConfig` and `collectCanonicalEvidence` disagree about the implicit cap. A shared constant (or single-source fallback) would keep behavior aligned across code paths.</comment>

<file context>
@@ -321,6 +321,14 @@ export function resolveVerifierConfig(
+        readChars(
+          env,
+          "VERIFIER_CANONICAL_EVIDENCE_CHARS",
+          4000,
+          truncDisabled,
+        ),
</file context>