Skip to content

Add heuristic fallback for skill selection when pr-triage sub-agent fails - #52708

Merged
pelikhan merged 4 commits into
mainfrom
copilot/deep-report-add-fallback-pr-triage
Aug 15, 2026
Merged

Add heuristic fallback for skill selection when pr-triage sub-agent fails#52708
pelikhan merged 4 commits into
mainfrom
copilot/deep-report-add-fallback-pr-triage

Conversation

Copilot AI commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Step 3 of mattpocock-skills-reviewer.md hard-depended on the pr-triage sub-agent returning parseable JSON, so a single flaky, timed-out, or malformed sub-agent call failed the entire run (success rate fell from 100% to 54.5%).

Changes

  • Step 3 fallback branch — on error, timeout, empty output, or unparseable JSON: retry at most once, log one line, then derive the selection locally. Explicitly does not abort.
  • Deterministic heuristic table — maps changed file paths and PR title/body to the same change_type / recommended_skills vocabulary the sub-agent emits. Rows are matched top-down, first match wins, with mixed_unclear/codebase-design + /tdd as the catch-all.
  • high_impact_files fallback — largest non-generated changed files by additions + deletions from the already pre-fetched pr-meta.json; key_signals empty. The review body notes when the fallback path was taken.
  • Step 4 wording — now reads "the Step 3 result" rather than naming pr-triage directly, so both paths flow through identically.
  • Recompiled mattpocock-skills-reviewer.lock.yml (body hash only; the prompt body is not embedded in the lock).

The heuristic reuses data already on disk, so the fallback adds no new tool calls or network dependencies.

| Signal in the changed file paths | `change_type` | `recommended_skills` |
| --- | --- | --- |
| Only `*.md`, `docs/**`, `.changeset/**` | `documentation` | `/grill-with-docs` |
| Only test files (`*_test.go`, `*.test.*`, …) | `tests_only` | `/tdd` |
| Title/body mentions fix/bug/regression/panic/crash | `bug_fix` | `/diagnosing-bugs`, `/tdd` |
|| | |
| Anything else | `mixed_unclear` | `/codebase-design`, `/tdd` |

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add fallback for pr-triage sub-agent call failure Add heuristic fallback for skill selection when pr-triage sub-agent fails Aug 14, 2026
Copilot AI requested a review from pelikhan August 14, 2026 12:05
@github-actions

Copy link
Copy Markdown
Contributor

PR Triage\n\n- Category: bug\n- Risk: low\n- Priority: medium\n- Score: 52/100 (impact 35 + urgency 10 + quality 7)\n- Recommended action: batch_review\n

Generated by 🔧 PR Triage Agent · auto · 62.8 AIC · ⌖ 2.76 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

🎯 Excellent Resilience Fix

Thanks for addressing the Matt Pocock Skills Reviewer reliability regression! This PR adds smart fallback logic when the pr-triage sub-agent times out or returns invalid JSON—directly solving the 54.5% success rate issue documented in #52519.

What I liked:

  • Deterministic heuristic — The file-path and title-matching logic reuses the same vocabulary as the sub-agent, ensuring consistent behavior whether the fallback triggers or not.
  • Zero new dependencies — The fallback reuses pre-fetched data already on disk (pr-meta.json), adding no network calls or tool overhead.
  • Clear documentation — The heuristic table in the PR body makes the fallback strategy immediately understandable.
  • Graceful degradation — Retries once, then proceeds with the local heuristic rather than aborting the entire run.

Coverage: The changes are tightly focused—updates to Step 3 wording and the lock file reflect the new logic without sprawl. A solid, targeted fix that should restore reliability without introducing new risk.

🚀 This looks ready for merge!

Generated by ✅ Contribution Check · auto · 55.5 AIC · ⌖ 4.15 AIC · ⊞ 8.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

  • Category: feature
  • Risk: low
  • Priority: low (score: 34/100)
  • Score breakdown: impact 12/50, urgency 8/30, quality 14/20
  • Recommended action: batch_review
  • Batch: small-features-tests-cluster

Automated triage by PR Triage Agent.

Generated by 🔧 PR Triage Agent · auto · 64.3 AIC · ⌖ 2.94 AIC · ⊞ 7.8K ·

@github-actions

Copy link
Copy Markdown
Contributor

PR Triage

Category: feature  |  Risk: low  |  Score: 26/100

Score breakdown

  • Impact: 12/50
  • Urgency: 8/30
  • Quality: 6/20

Recommended action: defer

Draft, no reviews yet, meta-tooling improvement — low urgency.

Generated by 🔧 PR Triage Agent · auto · 68.2 AIC · ⌖ 10.7 AIC · ⊞ 7.8K ·

@pelikhan
pelikhan marked this pull request as ready for review August 15, 2026 04:59
Copilot AI balanced review requested due to automatic review settings August 15, 2026 04:59
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a local heuristic fallback so triage failures no longer abort skills-based PR reviews.

Changes:

  • Adds fallback skill selection and high-impact file ranking.
  • Generalizes Step 4 to consume either triage or fallback results.
  • Recompiles workflow metadata.
Show a summary per file
File Description
.github/workflows/mattpocock-skills-reviewer.md Defines fallback triage behavior and heuristics.
.github/workflows/mattpocock-skills-reviewer.lock.yml Updates the generated body hash.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced


**Fallback — never fail the review because of triage.** If the `pr-triage` call errors, times out, returns empty output, or returns text you cannot parse as the documented JSON shape, do **not** retry more than once and do **not** abort. Log one line noting that triage was unavailable, then derive the selection yourself from `/tmp/gh-aw/agent/pr-meta.json` and `/tmp/gh-aw/agent/pr-diff.patch` using this heuristic:

| Signal in the changed file paths | `change_type` | `recommended_skills` |
Copilot AI and others added 2 commits August 15, 2026 05:11
…dd-fallback-pr-triage

# Conflicts:
#	.github/workflows/mattpocock-skills-reviewer.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts on this branch.

Resolved in 72b1b16 (merge) and 4ad6e3d (recompile). The only conflict was in the generated .lock.yml; the .md source merged cleanly and the lock file was regenerated from it.

Copilot AI requested a review from pelikhan August 15, 2026 05:14
@pelikhan
pelikhan merged commit b258efc into main Aug 15, 2026
@pelikhan
pelikhan deleted the copilot/deep-report-add-fallback-pr-triage branch August 15, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[deep-report] Matt Pocock Skills Reviewer: add fallback when pr-triage sub-agent call fails or returns invalid JSON

3 participants