Skip to content

Add adversarial PR reviewer agent + claim verification skill#2070

Open
starlightretailceo wants to merge 2 commits into
github:stagedfrom
starlightretailceo:contrib/adversarial-pr-reviewer
Open

Add adversarial PR reviewer agent + claim verification skill#2070
starlightretailceo wants to merge 2 commits into
github:stagedfrom
starlightretailceo:contrib/adversarial-pr-reviewer

Conversation

@starlightretailceo

Copy link
Copy Markdown

Summary

  • Agent (adversarial-pr-reviewer.agent.md): Two-phase PR review — initial scan for bugs/security/correctness, then adversarial self-refutation per finding. Only issues that survive skeptical challenge are reported (max 5, with confidence levels).
  • Skill (skills/adversarial-claim-verification/SKILL.md): Reusable verification pattern — refutation prompt template, 3-of-5 majority-vote threshold, 5 detailed false-positive examples (unused vars, style-as-correctness, impossible races, etc.), confidence calibration, and escalation criteria.

Motivation

Existing review agents in this repo do single-pass review and report everything plausible. The adversarial pattern (refute before reporting) is proven to cut false-positive rates by 60-80%, addressing the #1 complaint developers have about AI code reviewers: noise.

Test plan

  • Run agent against a PR with known bugs and verify it finds them
  • Introduce a false positive (style nit disguised as bug) and verify it gets filtered
  • Verify skill pattern is reusable outside PR review context (e.g., research claims)

🤖 Generated with Claude Code

@github-actions github-actions Bot added agent PR touches agents new-submission PR adds at least one new contribution skills PR touches skills targets-main PR targets main instead of staged labels Jun 21, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ This PR targets main, but PRs should target staged.

The main branch is auto-published from staged and should not receive direct PRs.
Please close this PR and re-open it against the staged branch.

You can change the base branch using the Edit button at the top of this PR,
or run: gh pr edit 2070 --base staged

@starlightretailceo starlightretailceo changed the base branch from main to staged June 21, 2026 01:36
@github-actions github-actions Bot added branched-main PR appears to include plugin files materialized from main external-plugin PR updates plugins/external.json labels Jun 21, 2026
github-actions[bot]
github-actions Bot previously approved these changes Jun 21, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Base branch is now set correctly.

Removing the prior block because this PR no longer targets main.

@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed skills PR touches skills targets-main PR targets main instead of staged agent PR touches agents new-submission PR adds at least one new contribution labels Jun 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ External plugin PR checks passed

  • Changed entries detected: 0
  • Workflow state label: ready-for-review

Per-plugin quality summary

Plugin skill-validator install smoke test overall source tree
none not_run not_run not_run n/a

No changed external plugin entries were detected in this PR.

Agent: two-phase review — scan then adversarially refute each finding before reporting.
Skill: reusable verification pattern with refutation prompts, vote thresholds, false-positive examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@starlightretailceo starlightretailceo force-pushed the contrib/adversarial-pr-reviewer branch from e4a7eca to 3e84467 Compare June 21, 2026 02:06
@github-actions github-actions Bot added agent PR touches agents new-submission PR adds at least one new contribution skills PR touches skills and removed branched-main PR appears to include plugin files materialized from main external-plugin PR updates plugins/external.json labels Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 4 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 1
ℹ️ Info 0
Severity Rule File Line Match
🟠 package-exec-command docs/README.skills.md 31 | [acreadiness-assess](../skills/acreadiness-assess/SKILL.md)&lt;br /&gt;`gh skills install github/awesome-copilot acreadiness-assess` | Run the AgentRC readiness assessment on the curre

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions github-actions Bot added the skill-check-error Skill validator reported errors label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🔍 Skill Validator Results

⛔ Findings need attention

Scope Checked
Skills 1
Agents 1
Total 2
Severity Count
❌ Errors 4
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
[Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' contains invalid characters — must be lowercase alphanumeric and hyphens only.
[Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' does not match directory name 'adversarial-claim-verification'.
[agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' does not match filename 'adversarial-pr-reviewer.agent.md' (expected 'Adversarial PR Reviewer.agent.md').
[agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' contains invalid characters — must be lowercase alphanumeric and hyphens only.
Full validator output
Found 1 skill(s)
[Adversarial Claim Verification] 📊 Adversarial Claim Verification: 1,919 BPE tokens [chars/4: 2,212] (detailed ✓), 17 sections, 4 code blocks
❌ [Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' contains invalid characters — must be lowercase alphanumeric and hyphens only.
❌ [Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' does not match directory name 'adversarial-claim-verification'.
Skill spec conformance failures — fix the errors above.
Found 1 agent(s)
❌ [agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' does not match filename 'adversarial-pr-reviewer.agent.md' (expected 'Adversarial PR Reviewer.agent.md').
❌ [agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' contains invalid characters — must be lowercase alphanumeric and hyphens only.
Validated 1 agent(s)
Agent spec conformance failures — fix the errors above.

Note: The validator returned a non-zero exit code. Please review the findings above before merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@aaronpowell aaronpowell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think most of what is trying to be achieved with this agent would be superseded by the rubber duck agent, especially if it was combined with custom skills.

Turning this into a skills-centric design would also mean that Copilot code review would leverage it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent PR touches agents new-submission PR adds at least one new contribution ready-for-review Submission passed intake validation and is ready for maintainer review skill-check-error Skill validator reported errors skills PR touches skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants