Skip to content

feat(sdk): surface verifierRejections across reports, CLI output, and findings-file export - #474

Open
CalebKAston wants to merge 13 commits into
getsentry:mainfrom
babylist:feat/upstream-verifier-rejections
Open

feat(sdk): surface verifierRejections across reports, CLI output, and findings-file export#474
CalebKAston wants to merge 13 commits into
getsentry:mainfrom
babylist:feat/upstream-verifier-rejections

Conversation

@CalebKAston

@CalebKAston CalebKAston commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

A finding the verification pass rejects simply disappears today — no count, no reason, visible nowhere in Warden's output. This adds a verifierRejections: {count, reasons[]} field, computed in verifyFindings() and threaded through postProcessFindings() onto SkillReport.

It's surfaced everywhere failedHunks/failedExtractions already are: the terminal report, the Reporter's aggregate summary (log + TTY modes), the JSONL run log (per-chunk and run-summary), and the GitHub Action's findings-file export. The run-level aggregate (JSONL summary, Reporter, terminal) intentionally surfaces only a count, not the reasons — matching the existing failedHunks/failedExtractions precedent. Reasons stay available at the per-skill chunk-record level.

Real-world verification

This has been running in production for a period feeding an internal data pipeline into an observability platform for tracking skill health. Querying that platform:

  • ~3,900 skill runs recorded in the window.
  • 555 of those runs (about 14%) logged at least one verifier rejection, totaling 874 rejected findings across three different review skills.
  • Spot-checking a sample of the recorded reasons shows the verifier doing real, substantive work, not rubber-stamping: the large majority cite a finding anchored outside the PR's actual changed files (out-of-scope), or reason through why a cited convention doesn't actually apply to the specific pattern flagged — exactly the "would have been noise for the author" cases this field exists to make visible.

Test plan

  • pnpm lint && pnpm build && pnpm test — all green (93 test files, 1839 passing, 4 pre-existing skips)
  • verify.test.ts — rejection counting, default-reason fallback, multi-finding aggregation
  • output.test.ts — findings-file export present/absent
  • terminal.test.ts, reporter.test.ts — per-skill and aggregate warning lines
  • jsonl.test.ts, main.test.ts — schema round-trip through the JSONL chunk stream and run summary
  • tasks.test.ts — end-to-end through the CLI task runner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant