From a957bf7d4889225d5012a1f1a095f8f7ada03f00 Mon Sep 17 00:00:00 2001 From: Shahzaib Date: Sat, 18 Jul 2026 12:31:20 -0700 Subject: [PATCH 1/4] Enhance Copilot Review Analyst skill: v3 newsletter report + adoption-signal & fairness methodology, Fixes AB#3696013 Report / rendering: - v3 newsletter Outlook report: 1200px hybrid-fluid shell, edge-to-edge newsletter masthead (inset hero + 1120px VML), big-number KPI hero grid, single-row meta-chips, Methodology section - Headline now leads with adoption % (Helpful & adopted), not precision - Document Outlook rendering learnings in report-formatting.md: single-row chip table (not inline-block), mso-line-height-rule:exactly for prose, edge-to-edge frame (whiten bgcolor AND inline CSS), inset masthead + VML width, Graph-draft delivery with Outlook COM fallback Classification / methodology: - Declined verdict + Copilot precision (Helpful / (Helpful + Incorrect)); Declined is neutral - Force-push confound validation before scoring precision - Engineer-intention nuance: correct-but-declined feedback is not a Copilot error - New adoption signals: reaction + apply-suggestion detection, merged-only PR filter, Phase 0 preflight coverage audit, review-coverage split, self-consistency gate, PR triage table Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d4fbe65f-9d71-4531-b63e-b5a12b21fba4 --- .../skills/copilot-review-analyst/SKILL.md | 45 +- ...e-Review-Effectiveness-Report-Outlook.html | 697 ++++++------- ...ilot-Code-Review-Effectiveness-Report.html | 962 +++++++++++++----- ...opilot-Code-Review-Effectiveness-Report.md | 348 ++++--- .../references/classification-rules.md | 107 +- .../references/report-formatting.md | 111 +- .../scripts/analyze.ps1 | 189 +++- .../scripts/append-history.ps1 | 133 +-- .../scripts/final-classification.ps1 | 180 +++- .../scripts/precise.ps1 | 25 +- .../scripts/preflight.ps1 | 201 ++++ .gitignore | 2 + 12 files changed, 2065 insertions(+), 935 deletions(-) create mode 100644 .github/skills/copilot-review-analyst/scripts/preflight.ps1 diff --git a/.github/skills/copilot-review-analyst/SKILL.md b/.github/skills/copilot-review-analyst/SKILL.md index cc10e63f..4832fdc9 100644 --- a/.github/skills/copilot-review-analyst/SKILL.md +++ b/.github/skills/copilot-review-analyst/SKILL.md @@ -1,6 +1,6 @@ --- name: copilot-review-analyst -description: Analyze GitHub Copilot code review effectiveness across Android Auth repositories. Collects all Copilot inline review comments via GitHub API, classifies them as helpful/not-helpful/unresolved through reply analysis, diff verification, and AI-assisted classification, then generates a report with per-repo and per-engineer statistics. Use this skill when asked to "analyze Copilot reviews", "measure Copilot review effectiveness", "generate Copilot review report", "how helpful are Copilot reviews", "run review analysis", or any request to measure/report on GitHub Copilot code review quality and adoption. +description: Analyze GitHub Copilot code review effectiveness across Android Auth repositories. Collects all Copilot inline review comments via GitHub API, classifies them as helpful/declined/incorrect/unresolved through reply analysis, diff verification, and AI-assisted classification, then generates a report with Copilot precision plus per-repo and per-engineer statistics. Use this skill when asked to "analyze Copilot reviews", "measure Copilot review effectiveness", "generate Copilot review report", "how helpful are Copilot reviews", "run review analysis", or any request to measure/report on GitHub Copilot code review quality and adoption. --- # Copilot Review Analyst @@ -57,7 +57,7 @@ What it does: 6. Record whether each comment has a reply and capture the reply text (no classification at this stage) **Outputs:** -- `$env:TEMP\copilot-review-analysis\raw_results.json` — all comments with `HasReply` flag and raw reply text +- `$env:TEMP\copilot-review-analysis\raw_results.json` — all comments with `HasReply` flag and raw reply text. Also records the **force-push fingerprint** per comment: `OriginalCommitId` (the immutable commit Copilot reviewed), `DiffHunk` (frozen copy of the lines Copilot saw), and `ReviewedCommitRewritten` (`true` when the reviewed commit is no longer in the PR's commit list — see [The Force-Push Confound](references/classification-rules.md#the-force-push-confound-mandatory-validation-before-scoring)). - `$env:TEMP\copilot-review-analysis\review_summaries.json` — PR-level summary comments (for reference) ### Phase 2: Diff-Level Verification @@ -87,22 +87,28 @@ What it does: - `file-not-changed` — file untouched after the comment - `no-subsequent-commits` — PR merged without any commits after the review +> **Force-push guard:** `precise.ps1` carries `ReviewedCommitRewritten` forward and flags any `not-applied`/`file-not-changed` verdict that sits on a rewritten reviewed snapshot. These are **unsafe to score as Not Helpful** until validated against the immutable `DiffHunk` — the "fix" may live in the orphaned commit. The script prints a `FORCE-PUSH CONFOUND` warning listing them. Resolve each in Phase 3 before finalizing. + **Output:** `$env:TEMP\copilot-review-analysis\precise.json` ### Phase 3: AI Reply Classification This phase is performed by the agent (you) in conversation. Classify **every replied comment** by reading the full Copilot comment and engineer reply in context. -See [references/classification-rules.md](references/classification-rules.md) for detailed guidance on what counts as helpful vs not-helpful. +See [references/classification-rules.md](references/classification-rules.md) for detailed guidance on the three-way replied-comment verdict. **Process:** 1. Load `raw_results.json` and filter to `HasReply -eq true` 2. For each comment, read the `CommentBody` (Copilot's feedback) and `HumanReplyText` (engineer's reply) -3. Classify as `helpful` or `not-helpful` based on the engineer's intent (read the full reply, don't keyword-match) -4. Also review `file-changed-elsewhere` and `file-changed-no-line-info` verdicts from Phase 2 to identify re-audit flips +3. Classify each reply as `helpful`, `declined`, or `not-helpful` (incorrect). Read the full exchange; do not keyword-match. + - `helpful`: feedback was accepted, fixed, or acknowledged as actionable + - `declined`: feedback was correct/reasonable, but the engineer intentionally chose not to act (by-design, subjective, contextual, moot, or out-of-scope). This is neutral and does not count against Copilot. + - `not-helpful`: the reply demonstrates Copilot was factually or technically wrong. This is the only replied verdict that counts against Copilot quality. +4. **Force-push validation pass (do this before finalizing any `declined`/`not-helpful`/`unresolved`):** for every such comment with `ReviewedCommitRewritten = true`, the engineer rewrote the commit Copilot reviewed — an "outdated / already there / not relevant" dismissal may be hiding a comment that was *correct* at the reviewed snapshot. Re-derive Copilot's claim from the immutable `DiffHunk` (and, if needed, `contents?ref=`) and compare to the merged state. Flip to `helpful` if the merged code already satisfies the claim; keep `declined` only if declined on the merits; assign `not-helpful` **only** if Copilot was wrong *at `OriginalCommitId`*. Never score a rewritten-snapshot comment as `not-helpful` without that confirmation. Full rule + reproducible `gh` commands in [The Force-Push Confound](references/classification-rules.md#the-force-push-confound-mandatory-validation-before-scoring). +5. Also review `file-changed-elsewhere` and `file-changed-no-line-info` verdicts from Phase 2 to identify re-audit flips **Outputs** (write to `$env:TEMP\copilot-review-analysis\`): -- `reply-verdicts.json` — `{ "commentId": "helpful"|"not-helpful", ... }` for every replied comment +- `reply-verdicts.json` — `{ "commentId": "helpful"|"declined"|"not-helpful", ... }` for every replied comment - `reaudit-flips.json` — `{ "reauditFlipKeys": ["repo/prNum/filePattern", ...] }` for no-reply comments with strong evidence See `references/manual-audit-template.json` for the schema. @@ -157,7 +163,7 @@ Append a snapshot of the current run to the persistent history file for trend tr What it does: 1. Load `final_classification.json` and `precise.json` -2. Compute aggregate stats: response rate, three-way breakdown, per-repo, per-engineer +2. Compute aggregate stats: response rate, four-way report breakdown, Copilot precision, per-repo, per-engineer 3. Compute normalized volume: `comments/week` for fair comparison across different period lengths 4. Load existing `history.json` (or create empty if first run) 5. Append current snapshot; replace if same period already exists @@ -178,7 +184,7 @@ Generate both Markdown and Outlook-compatible HTML reports. **Style/structure references** (in `assets/` — these contain data from the Jan-Mar 2026 analysis and serve as structural templates, NOT to be copied verbatim): - `assets/Copilot-Code-Review-Effectiveness-Report.md` — Markdown reference (~270 lines, ~3300 words) -- `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html` — Outlook HTML reference (~430 lines, ~42KB) +- `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html` — Outlook HTML reference (~345 lines, ~86KB). This is the current **v3 newsletter** design: 1200px hybrid-fluid shell, edge-to-edge newsletter masthead, big-number KPI hero grid, single-row meta-chips, and a Methodology section. It is the **anonymized** sample (uses "Engineer A") and is written single-quoted (escaping-free) for a trivial Graph push. - `assets/Copilot-Code-Review-Effectiveness-Report.html` — Standard HTML reference **Important:** The asset templates contain hardcoded numbers (557 comments, specific percentages, engineer names, etc.) from the first analysis. For each new run, generate fresh reports using the same section structure and formatting patterns but with statistics computed from `final_classification.json`. @@ -201,10 +207,16 @@ Generate both Markdown and Outlook-compatible HTML reports. After generating each report, compare its dimensions against the template: - **Markdown:** Must be ≥250 lines and ≥3000 words. If significantly smaller, the report is missing narrative depth. -- **Outlook HTML:** Must be ≥400 lines and ≥35KB. If significantly smaller, the report is missing visual elements or prose. +- **Outlook HTML:** Must be ≥300 lines and ≥70KB. If significantly smaller, the report is missing visual elements or prose. (The v3 newsletter design is single-quoted and layout-dense, so it runs fewer lines but far more KB than the old ``-based template.) If a report doesn't meet these thresholds, re-read the template and identify what's missing before saving. +#### v3 Newsletter Design & Delivery (Outlook) + +- **Headline leads with adoption.** The hero band and the first Key Takeaway lead with **Helpful & adopted %** (e.g. "78.3% of comments adopted"), with precision as the supporting second number — never lead with precision (it is near-100% by construction and overstates the story). +- **Load-bearing render techniques** live in [references/report-formatting.md](references/report-formatting.md) § *v3 Newsletter Rendering*: single-row meta-chip table (not `inline-block` tables — Outlook stacks them), `mso-line-height-rule:exactly` on body prose, edge-to-edge frame (whiten `bgcolor` **and** inline CSS), and the inset masthead + `1120px` VML width. Apply the edge-to-edge/masthead transforms report-scoped — never edit the shared shell. +- **Deliver as a color-preserving draft**, not paste. Preferred: Graph draft (`workiq-create_entity` on `/me/messages`). Fallback when WorkIQ/Graph is unavailable: Outlook COM (`Outlook.Application` → set `.HTMLBody` → `.Save()`/`.Display()`) with an ASCII-safe subject. + **Generate two versions of each report:** 1. **Team-internal** — uses real engineer names (from account map). For the team. 2. **Org-wide** — anonymizes engineers as "Engineer A", "Engineer B", etc., sorted by helpfulness descending. For sharing outside the team. @@ -212,8 +224,8 @@ If a report doesn't meet these thresholds, re-read the template and identify wha **Process:** 1. Load `final_classification.json` 2. **Read full template files** (`assets/Copilot-Code-Review-Effectiveness-Report.md` and `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html`) — do NOT skip this step -3. Compute aggregate statistics (total, per-repo, per-engineer, three-way breakdown) -4. Load `raw_results.json` to collect full comment text for 4-5 helpful and 4-5 unhelpful examples +3. Compute aggregate statistics (total, per-repo, per-engineer, four-way breakdown, precision) +4. Load `raw_results.json` to collect full comment text for helpful, declined, and genuinely incorrect examples 5. Load `~/.copilot-review-analysis/history.json` for trend data. If ≥2 entries exist, generate a **Trend** section comparing the current run with the previous run(s). See [references/report-formatting.md](references/report-formatting.md) for trend section formatting rules. 6. Generate reports matching the template's section structure, narrative depth, and visual formatting 7. **Verify dimensions** (word count, line count) against the quality gate thresholds @@ -227,12 +239,17 @@ See [references/report-formatting.md](references/report-formatting.md) for the r ## Key Principle: "Unresolved" ≠ "Not Helpful" -Comments with no reply and no diff evidence are **Unresolved**, not assumed unhelpful. This is a critical distinction: +The report must present four distinct outcome buckets: - **Confirmed Helpful** = positive evidence (explicit acknowledgment OR verified fix in diff) -- **Confirmed Not Helpful** = positive evidence (explicit dismissal with stated reason OR comment on stale code) +- **Declined** = Copilot was correct/reasonable, but the engineer intentionally chose not to act. This is neutral. +- **Incorrect** = a replied comment where the engineer demonstrated that Copilot was factually or technically wrong - **Unresolved** = insufficient evidence either way (engineer never engaged) -The `final-classification.ps1` script classifies no-response/no-diff-evidence comments as "not-helpful" for conservative stats, but the report should present the three-way breakdown to be honest about uncertainty. +Comments with no reply and no diff evidence are **Unresolved**, not assumed incorrect. Internally, `final-classification.ps1` retains the `not-helpful` verdict for these conservative no-reply cases; the report must split that value by `Replied`: replied `not-helpful` is Incorrect, while unreplied `not-helpful` is Unresolved. + +Report **Copilot precision** as `Helpful / (Helpful + Incorrect)`. Exclude Declined and Unresolved from both numerator and denominator. + +**Compute precision only after the force-push validation pass.** A comment on a rewritten snapshot can be a correct comment masquerading as a false positive, and it lands directly in the precision denominator. Before finalizing precision, resolve every `ReviewedCommitRewritten = true` comment per [The Force-Push Confound](references/classification-rules.md#the-force-push-confound-mandatory-validation-before-scoring), and report the count of rewritten-snapshot comments as a measurement-integrity note so readers know the denominator was validated. ## Copilot Comment Identification diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html index 4963fed1..c03b2f5d 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html @@ -1,432 +1,345 @@ - + - - + + + +Copilot Code Review Effectiveness -Copilot Code Review Effectiveness Analysis + -
+ -

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

+ +
Copilot review effectiveness - engineers adopted 78% of 138 comments; 99.1% precision, only 1 genuine error.
- -

 

Background
+ + -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

+ + + -

2. Review before merging. 38% of ignored comments were on the final commit — the engineer merged without looking at Copilot's last round. Building a habit of checking review comments before clicking merge would give this feedback a chance to have impact.

+ + -

3. Improve the review instructions. The most common dismissal reasons — "lacks domain context," "suggests tests for trivial code," "misunderstands our APIs" — are things we can address by refining our copilot-instructions.md. Adding guidance like "don't suggest tests for telemetry-only changes" or providing context about specific domain patterns could reduce the noise and make the signal-to-noise ratio better for everyone.

+
-

The Android Auth Platform team builds and maintains the authentication libraries used by Microsoft's mobile apps — including MSAL (client-side), Common (shared IPC and utilities), and Broker (the brokered authentication service running on Android devices). Our codebase spans three repositories with 10 active contributors.

+ + -

Earlier this year, we enabled GitHub Copilot code reviews across all three repositories. We also added custom review instructions via copilot-instructions.md to give the AI context about our architecture, coding conventions, and multi-repo structure — aiming to make its feedback more relevant than generic suggestions.

- -

This report answers the question: Is Copilot code review actually useful for our team? We looked at every comment Copilot left over 2 months and determined — through reply analysis, commit diff verification, and AI-assisted classification of ambiguous cases — whether each piece of feedback led to a real code improvement, was dismissed as irrelevant, or was simply never evaluated by the engineer.

+ + - -

 

At a Glance
+ +
+ + + + + + + + + + +
Window 5 weeks Comments 138 Helpful & adopted 78.3% Precision 99.1% View Org-wide (anonymized)
+
- - - - - - - - +
- - -
-
57%
-
of comments received
no response
-
-		 - - -
-
41%
-
confirmed
helpful
-
-		 - - -
-
18%
-
confirmed
not helpful
-
-		 - - -
-
41%
-
unresolved
(inconclusive)
-
-
+
What's New This Cycle
- - - - - - +
Three things that stand out this period
+
 
+
This is our fourth measurement window. Three things stand out from the data.
+
 
+
1. Engineers adopted 78.3% of Copilot's comments outright — a new high. On the code that actually shipped, only 2.2% were left unevaluated. Crucially, 23 adoptions came with no reply and were confirmed against the merged diff (13 by matching Copilot's exact suggestion to the fix commit) — so silent adoption is credited, not lost.
+
 
+
2. We measure what actually merged — plus coverage. The corpus is the 138 comments on human PRs that merged in-window, so every figure describes shipped code. Copilot reviewed 86.4% of merged human PRs (70 of 81). This “what-shipped” basis is a deliberate change from prior cycles, so the rates aren't directly comparable to earlier periods (see Trend).
+
 
+
3. Two numbers — and helpfulness leads. Helpfulness (78.3%) is the headline: how often engineers adopted a comment. Precision (99.1%) backs it up: only one of 138 comments was a genuine error. The gap between them is deliberate declines (18.8%) — correct feedback an engineer consciously chose not to take, not Copilot mistakes.
+
 
+
-The biggest finding isn't about AI quality — it's about adoption. -Of the 557 comments Copilot left, only 239 (43%) received any response from an engineer — the other 318 were never acknowledged. Of those 239 responded-to comments, 144 (60%) led to confirmed code improvements. That's a strong signal: when engineers read and evaluate Copilot's feedback, the majority of it is useful. The challenge is that 57% of comments are never evaluated at all, leaving 226 (41%) in an unresolved state where we can't determine whether they were helpful or not. -
+
At a Glance
- -

Scope

- - - - - - - - +
Helpfulness leads — precision backs it up
+
 
+
We analyzed every inline Copilot comment on the human-authored PRs that merged this period — 138 comments on shipped code.
+
 
+
- - -
-
163
-
Human PRs
scanned
-
-		 - - -
-
113
-
PRs received
Copilot review (69%)
-
-		 - - -
-
557
-
Inline review
comments
-
-		 - - -
-
4.9
-
Avg comments
per reviewed PR
-
-
+ + + + + + + +
+
+
78.3%
+
Confirmed helpful & adopted
+
108 of 138 comments engineers acted on
+
+ 		+
+
99.1%
+
Review precision
+
Only 1 of 109 judged comments was wrong
+
+
+
+
1
+
Genuine error — 0.7%
+
The only slice that reflects on the AI
+
+ 		+
+
81.2%
+
Engineer response rate
+
112 of 138 comments engaged
+
+
- - -

 

Overall Results
- -

Engineer Response Rate

-

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- - - - - - - +
 
+
43% replied57% ignored
+ +
+
How to read this
+
Two numbers, two questions. Helpfulness = did we act on it? (Helpful ÷ all comments = 108 ÷ 138 = 78.3%). Precision = when judged, was Copilot right? (Helpful ÷ (Helpful + Incorrect) = 108 ÷ 109 = 99.1%). A comment can be precise but not helpful — that's the Declined bucket, where Copilot was correct but the engineer made a deliberate call. Of comments engineers actually replied to, ~76% were adopted (85 ÷ 112); the rest were considered declines. That gap is judgment, not error.
+
+
 
+ +
Helpfulness Verdict
- - - - - - - +
The full four-way breakdown
+
 
+
Every comment lands in one of four buckets. Note the red Not helpful segment is barely visible at 0.7% — the only slice that counts against Copilot.
+
 
+
Jun 10 – Jul 15 · 138 comments
 
Engineer replied (239)
 
No response (318)
Helpful 78.3%Declined 18.8%  
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+ + +
+
Formula
+
Precision = Helpful ÷ (Helpful + Incorrect) = 108 ÷ (108 + 1) = 99.1%. Declined and Unresolved are excluded from both sides — neither is a Copilot mistake — so we never penalize the AI for feedback an engineer simply chose not to take.
+
+
 
+ +
By Repository
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- - - - - - - - +
Every repo clears 97% precision
+
 
+
41.3% Helpful18.1%40.6% Unresolved
RepositoryCommentsResponseHelpfulDeclinedIncorrectUnresolvedPrecision
Broker6978.3%52 (75.4%)15 (21.7%)0 (0.0%)2 (2.9%)100%
Common5680.4%45 (80.4%)9 (16.1%)1 (1.8%)1 (1.8%)97.8%
MSAL13100%11 (84.6%)2 (15.4%)0 (0.0%)0 (0.0%)100%
+
 
+
MSAL leads on response (100%) and helpfulness (84.6%) with zero incorrect comments. Broker carries the highest volume and the largest Declined bucket — a team that engages heavily and makes deliberate design calls — and still posts a clean 100% precision. The Incorrect column — the only one that reflects poorly on the AI — is a single comment across all three repos (in Common).
+
 
+ +
By Engineer
- - - - - - - - - +
Declined is not a failure
+
 
+
 
Confirmed Helpful (230)
 
Confirmed Not Helpful (101)
 
Unresolved (226)
EngineerCmtsRespHelpfulDeclinedIncorrectUnresPrecision
Engineer A3070%28200100%
Engineer B2282%14602100%
Engineer C2167%12801100%
Engineer D1573%13200100%
Engineer E14100%12200100%
Engineer F1182%10100100%
Engineer G8100%7100100%
Engineer H5100%401080%
Engineer I5100%3200100%
Engineer J4100%2200100%
Engineer K3100%3000100%
+
 
+ + +
+
Insight
+
The Declined column changes how we read this table. Under the old binary rubric, Engineer C (8 declined) and Engineer B (6 declined) would have looked like the engineers getting the least value from Copilot. In fact both post 100% precision — Copilot was never actually wrong on their PRs. They engage heavily and make frequent, deliberate design-tradeoff calls. Engagement and judgment, not accuracy, are what these numbers really track.
+
+
 
+ +
Trend: Historical Comparison
- - - - - - - - - - - - - - - - - - - - - - - - - - - +
This period vs previous — read across a basis change
+
 
+
One caution before the numbers: this cycle changed how we count. We scoped to merged PRs and credited silent adoption; prior periods counted all comments by creation date. So the quality metrics aren't apples-to-apples — read the pre-June rows as directional only.
+
 
+
Jun 10 – Jul 15 (current basis)
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. All 226 are comments where the engineer did not respond. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
Helpful 78.3%Declined 18.8%  
+
 
+
Apr 20 – Jun 9 (previous basis)
Helpful 52.9%16.6%Unresolved 30.6%
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+
Current vs previous period. Two things you can read cleanly across the basis change: response rate is genuinely up, and precision — the one metric that isolates AI quality — exists from this cycle forward. Rows driven by the basis shift are marked.
+
 
+
MetricPrev · 50dThis · 35dChange
Response rate67.5%81.2%▲ +13.7pp
Replied-helpful rate75.5%75.9%≈ flat
Helpful (of all)52.9%78.3%+25.4pp · basis
Unresolved30.6%2.2%−28.4pp · basis
Copilot precisionn/a99.1%new
+
 
+
The full multi-cycle run — the “Dismissed” column combines Declined + Incorrect for continuity, and the current row (highlighted) is the only one on the merged what-shipped basis:
+
 
+
PeriodDaysCommentsCmts/wkResponseHelpfulDismissedUnresolved
Jan 24 - Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 - Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 - Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 - Jul 153513827.681.2%78.3%19.6%2.2%
+
 
+
Historical “not helpful” rates of 16–21% almost certainly overstated how often the AI was actually wrong — they bundled deliberate declines with genuine errors. Treat pre-June quality as an upper bound, not a fact. The step change at June is largely a definition change, not a behavior change.
+
 
+ +
What Copilot Is Good At
- - - - - - +
Real catches from this period
+
 
+
-226 comments (41%) are unresolved — not because the AI was wrong, but because no one engaged. -All 226 are comments that received no reply from the engineer. This includes 122 comments on the final commit before merge (where the engineer had no subsequent commits), 45 where the file was modified but at different lines, 50 where the comment had no line number for verification, and 9 where the file was never modified. If engineers had responded — even just to dismiss — we would know whether the feedback was useful. -
+
+
Copilot flagged that a flight/telemetry provider was being read before it was initialized — an ordering bug that would silently drop data. “Good catch — this is a fundamental ordering problem. Fixing it.”
+
— Broker PR #215 · AriaInitializer
+
- -

How Helpful Comments Were Delivered

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
+
+
A newly-added overload shadowed a deprecated one; Copilot recommended @JvmOverloads to preserve binary compatibility for Java callers. “Fixed. Thanks for pointing this out.”
+
— Common PR #3147
+
- -

How Not Helpful Comments Were Identified

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
+
+
On an OWASP dependency check, Copilot spotted an inconsistency in how a CVE suppression was applied across modules. “Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951.”
+
— Common PR #3156
+
- - -

 

Results by Repository
- - -

Broker (293 comments)

- - - - - - +
 
+
49%16%35%
+
When Copilot Was Actually Wrong
- - -

Common (188 comments)

- - - - - - +
The one genuine error, in full — all 0.7%
+
 
+
36%18%46%
+
+
Copilot warned a Mockito varargs matcher would misbehave. “Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12.” A version-specific false positive — the engineer's repro proved Copilot wrong. This is the only genuine error in all 138 comments.
+
— Common PR #3171
+
- - -

MSAL (76 comments)

- - - - - - +
 
+
26%25%49%
+ +
+
Measurement integrity
+
The force-push confound. A prior draft nearly counted a second error: a correct Broker @Override comment (PR #212) almost recorded as a hallucination because the engineer force-pushed a rewritten commit ~2 min after opening the PR. Copilot had reviewed the original commit, where the annotation was genuinely absent; the force-push orphaned it, so GitHub marked the note “outdated.” We found this fingerprint on 31 of 138 comments (9 PRs), but only #212 was a proven wrongful-outdating — so 99.1% precision holds. Every “already there / outdated” dismissal on a force-pushed PR is now validated against the immutable reviewed snapshot before it's scored.
+
+
 
+ +
When Copilot Was Right but Declined
- - - - - - - - - - +
Correct feedback, deliberate call — neutral
+
 
+
We hand-audited all 26 Declined comments. Two things stand out: none were Copilot errors — every one was a correct or reasonable note — and all 26 were actively replied to. Each is a deliberate, reviewed call where the engineer read the note and chose otherwise.
+
 
+
 
Helpful
 
Not Helpful
 
Unresolved
+
+
Copilot flagged the comment ~1 per device as potentially misleading. “~ = approximate. i think this is ok.” A fair point the author weighed and consciously declined — a judgment call another reviewer might have taken the other way. Not a Copilot error.
+
— Broker PR #201 · the flagship case
+
- - - - - - - - - - - +
 
+
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
+
+
Copilot suggested not logging a full stack trace for an expected exception. “Since this is a new feature, we want that stack trace for all exceptions.” A sound general heuristic that simply did not apply here.
+
— Broker PR #223
+
- - -

 

Results by Engineer
-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- - - - - - - - - - - - - - - - - - - - - - +
 
+
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
+
What We'd Recommend
- - - - - +
Three opportunities
+
 
+
-Engagement drives value — and visibility. -Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
+ + + +
+
1
+ 		+
Keep engaging — it's working
+
Response rate rose to 81.2% (from 67.5%), and engineers adopted 23 comments silently. A quick reply — even a decline with a one-line reason — is what lets us separate “Copilot was wrong” from “I chose otherwise”; silent adoption we now credit automatically.
+
- - -

 

What Copilot Is Good At
- -

Catching real bugs:

-
-PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers.
-Engineer reply: "You're right. Making the change." -
- -

Stale documentation and naming inconsistencies:

-
-PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant after it was renamed. All four were silently fixed in the next commit. -
- -

CI/pipeline configuration issues:

-
-PR #3038 (Common): Copilot warned that using vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. -
- -

The @copilot apply workflow:

-
-In 16 instances (2.9%), engineers validated the feedback and delegated the fix back: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue. -
- - -

 

What Copilot Struggles With
- -

Lacking domain context:

-
-Copilot: "shared_device_id could be used for tracking — consider hashing before emission."
-Engineer: "The shared_device_id is a random UUID and is not PII." -
- -

Suggesting tests for trivial code:

-
-Copilot: "New telemetry attributes lack test coverage..."
-Engineer: "These are just telemetry related changes and adding unit tests will be overdo here." -
- -

Misunderstanding APIs:

-
-Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."
-Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." -
- - -

 

Key Takeaways
- - - - - - - - - - +
 
+
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality, the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
+ + + +
+
2
+ 		+
Validate “outdated” dismissals before scoring them
+
The force-push confound produced one near-miss this cycle: a correct comment (Broker #212) was almost recorded as a hallucination because a rebase orphaned the reviewed commit. We now validate every “already there / outdated” dismissal on a force-pushed PR against the immutable reviewed snapshot. Engineers can help by not rebasing away the reviewed commit mid-review.
+
- - -

 

How We Measured This
- - - - - - - - +
 
+
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used GitHub compare API to check if suggestion tokens appeared as diff additions, or if exact line ranges were modified.
4. AI-assisted classificationAll 133 ambiguous replies were read and classified by GitHub Copilot, based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong.
+ + + +
+
3
+ 		+
Tune copilot-instructions.md against real false-positive patterns
+
Only one genuine error slipped through (a Mockito 5.11.0 version assumption). The higher-leverage fix is the recurring declined-but-correct nitpicks: teach Broker that telemetry attribute values can be intentional literals, stop flagging approximate “~” rationale comments, and have all repos re-verify a missing @Override/test/#TBD against the PR head before flagging.
+
+
 
+ + +
How We Measured This
+
Methodology notes
+
 
+
Corpus. Every inline Copilot review comment on human-authored PRs that merged between June 10 and July 15, 2026 — 138 comments across Common, MSAL, and Broker. Bot-authored PRs and comments on unmerged or abandoned PRs are excluded, so every number describes code that actually shipped.
+
 
+
Coverage. Copilot reviewed 86.4% of merged human PRs (70 of 81). Most uncovered PRs are trivial (No-Changelog) or predate Copilot review being enabled on that repo.
+
 
+
Four-way verdict. Each comment is hand-classified into exactly one bucket: Helpful (adopted), Declined (correct or reasonable, but the engineer made a deliberate call not to take it), Incorrect (Copilot was factually wrong), or Unresolved (never evaluated). Declined and Incorrect are separated on purpose — only Incorrect counts against the AI.
+
 
+
Silent adoption. Adoption isn't read from replies alone. Where an engineer changed the code without replying, we verify against the merged diff — 13 of 23 silent adoptions were confirmed by matching Copilot's exact suggestion to the fix commit. The diff verification itself is long-standing; what changed this cycle is that we now credit these as Helpful rather than leaving them Unresolved.
+
 
+
Precision. Precision = Helpful ÷ (Helpful + Incorrect). Declined and Unresolved are excluded from both sides, so the metric answers only “when Copilot was judged, was it right?” and never penalizes the AI for feedback an engineer simply chose not to take.
+
 
+
Force-push validation. For every “already there / outdated / not relevant” dismissal we check whether the reviewed commit was later rewritten (original_commit_id vs the PR's current commit list). If it was, we re-derive Copilot's claim from the immutable diff_hunk against the merged state before scoring — so a rebase can't silently record a correct comment as an error (see Broker #212).
+
 
+
Trend caveat. Pre-June periods counted all comments by creation date under a binary rubric; this cycle scopes to merged PRs with the four-way verdict. Cross-period quality comparisons are directional only.
+
 
+
Anonymization. The org-wide edition replaces engineer names with stable Engineer A–K labels ordered by comment volume; all counts are otherwise identical to the team edition.
-

Classification rules: "Confirmed Helpful" requires positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" requires positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage are classified as "Unresolved" rather than assumed unhelpful.

- - -

 

What We'd Recommend
- -

Based on this analysis, we see three opportunities:

-

1. Engage with the feedback. The single biggest improvement would be for engineers to respond to Copilot's review comments — even if just to dismiss them. When engineers engage, 60% of comments are helpful. When they don't, 71% of comments become a black hole of unknown value. A quick "not applicable" is more useful than silence — it tells us whether the AI is giving good feedback, and it builds the data we need to improve the review instructions.

+
+ Generated by the copilot-review-analyst skill · rendered with outlook-highfidelity-report · Merged what-shipped basis (PRs merged Jun 10 – Jul 15, 2026). Methodology: four-way verdict (Helpful / Declined / Incorrect / Unresolved) + Copilot precision, with silent-adoption crediting and force-push-snapshot validation. +
+ - - -
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

- -
+ - - - - - - - - - - - - - - - - - diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html index 72759b00..0fa1f846 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html @@ -98,314 +98,746 @@ .card-grid { grid-template-columns: repeat(2, 1fr); } .donut-container { flex-direction: column; } } + /* --- 4-way methodology additions --- */ + :root { + --blue: #0969da; --light-blue: #ddf4ff; + --purple: #8250df; --light-purple: #fbefff; + } + .card.blue { border-left: 4px solid var(--blue); } + .card.purple { border-left: 4px solid var(--purple); } + .bar-fill.blue { background: var(--blue); } + .bar-fill.purple { background: var(--purple); } + .stacked-segment.green { background: var(--green); } + .stacked-segment.blue { background: var(--blue); } + .stacked-segment.red { background: var(--red); } + .stacked-segment.amber { background: var(--amber); } + .card.six-grid { } + .card-grid.six { grid-template-columns: repeat(6, 1fr); } + @media (max-width: 640px) { .card-grid.six { grid-template-columns: repeat(2, 1fr); } }

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

- - -

Executive Summary

- -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

- -
-
-
57%
-
of comments received
no response
-
-
-
41%
-
confirmed
helpful
-
-
-
18%
-
confirmed
not helpful
-
-
-
41%
-
unresolved
(not evaluated)
-
-
- -
- The biggest finding isn't about AI quality — it's about adoption. - When engineers do engage with Copilot's comments, 60% turn out to be helpful. But 57% of comments receive no response at all, and 41% of all comments are unresolved — we can't tell if they were useful because no one evaluated them. The true helpfulness rate lies between 41% (confirmed floor) and 82% (if all unresolved were helpful). -
- - -

Scope

- -
-
-
163
-
Human PRs
scanned
-
-
-
113
-
PRs received
Copilot review (69%)
-
-
-
557
-
Inline review
comments
-
-
-
4.9
-
Avg comments
per reviewed PR
-
+

Android Auth Platform  |  June 10 – July 15, 2026  |  Common, MSAL, Broker Repositories

+ +

Executive Summary

+

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the 132 comments, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because Copilot was wrong or because the engineer made a deliberate judgment call.

+
+
72%
engineer
response rate
+
54.5%
confirmed
helpful
+
20.5%
correct but
declined
+
1.5%
actually
incorrect
+
23.5%
unresolved
(no evidence)
+
97.3%
review
precision
- - -

Overall Results

- -

Engineer Response Rate

+
Fairness principleWe separate “Declined” (Copilot was correct or reasonable, but the engineer intentionally chose not to act — a nitpick, a stylistic preference, or an approximation the author knew about) from “Incorrect” (Copilot was factually wrong). Only genuinely Incorrect comments count against review quality. Precision = Helpful ÷ (Helpful + Incorrect), excluding Declined and Unresolved — so we never penalize Copilot for feedback it had no way to know the author would decline.
+

This run introduces a new, fairer methodology. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral Declined category and reserves Not Helpful exclusively for comments where Copilot was demonstrably wrong.

+

Key findings:

+
    +
  • 72% of Copilot's comments now receive a response from engineers. This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value.
    • +
  • Only 1.5% of all comments were genuinely wrong. Across 132 comments, just two were confirmed as Copilot errors. When you compute Copilot's precision — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at 97.3%. The AI is rarely incorrect.
    • +
  • 20.5% of comments were correct-but-declined. These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. These do not count against Copilot's quality. Treating them as failures, as the old methodology did, materially understated the tool's accuracy.
    • +
  • 54.5% of all comments led to a confirmed code improvement. With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy.
    • +
+
+

How We Measured This

+

This analysis went through five phases to ensure accuracy:

+
    +

  1. Data collection. We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text.

    +
    2. +

  2. Diff-level verification. For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit.

    +
    3. +

  3. AI-assisted reply classification (three-way). For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts:

    +
      +
    • Helpful — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot).
      • +
    • Declined — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). Copilot was not wrong — it simply had no way to know the author's intent.
      • +
    • Not Helpful (incorrect) — the engineer's reply demonstrates the comment was factually or technically wrong (a false positive, a hallucinated problem, or a misunderstanding of a library/API).
      • +
    +
    4. +

  4. The fairness principle. The dividing line between Declined and Not Helpful is whether Copilot was wrong, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is Declined, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as Not Helpful.

    +
    5. +

  5. Cross-validation. All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved.

    +
    6. +
+

The final dataset classifies each of the 132 comments as confirmed helpful, declined (correct but not adopted), not helpful (incorrect), or unresolved (insufficient evidence to determine). No comment is left without a classification.

+
+

A Note on Fairness: Why "Declined" Is Not "Wrong"

+

The single most important change in this report is the recognition that an engineer declining a comment is not evidence that the comment was bad.

+

Consider Broker PR #201. Copilot flagged a code comment that read ~1 per device as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: "~ = approximate. i think this is ok." Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair.

+

We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise.

+

By separating these Declined cases (20.5% of all comments) from genuine errors (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually wrong in only 2 of 132 comments.

+
+

Overall Results

+ + + + + + + + + + + + + + + + + + + + + + + + + +
MetricValue
Human PRs scanned87
PRs that received Copilot review53 (61%)
Total inline review comments132
Average comments per reviewed PR2.5
+

Engineer Response Rate

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- -
-
43% replied
-
57% ignored
-
-
- Engineer replied (239) - No response (318) -
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- + + + + + + + + + + + + + + + + + + + + +
BehaviorCountPercentage
Engineer replied (any response — acceptance, decline, or discussion)9572.0%
Engineer did not reply3728.0%
+

At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a 100% response rate across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted.

+

Helpfulness Verdict

+

Each comment was classified into one of four categories. Note the deliberate split between Declined (Copilot was right, the engineer chose otherwise) and Not Helpful (Copilot was wrong):

-
41.3%
-
18.1%
-
40.6%
+
54.5%
+
20.5%
+
+
23.5%
- Confirmed Helpful (230) - Confirmed Not Helpful (101) - Unresolved (226) + Helpful (72) + Declined (27) + Incorrect (2) + Unresolved (31)
- - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
VerdictCountPercentageDefinition
Confirmed Helpful7254.5%The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Declined (correct, not adopted)2720.5%The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. This is neutral; Copilot was not wrong.
Not Helpful (incorrect)21.5%The engineer demonstrated the comment was factually or technically wrong — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality.
Unresolved3123.5%The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated.
- -
- 226 comments (41%) are unresolved — not because the AI was wrong, but because no one looked. - This includes 122 comments on the final commit before merge where the engineer had no subsequent commits, 45 where the file was modified at different lines, 50 with no line number for verification, and 9 where the file was never modified. If engineers had engaged — even just to dismiss — we would know. -
- -

How Helpful Comments Were Delivered

+

The Precision Metric

+

Because Declined is neutral and Unresolved is unknown, the fairest single measure of Copilot's review quality is its precision — of the comments we can actually adjudicate as right or wrong, what fraction were right?

+
+

Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%

+
+

Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. 97.3% precision means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy.

+

How Each Category Breaks Down

+

Confirmed Helpful (72):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
PathCountDescription
Engineer replied and acknowledged66Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …")
Engineer silently applied the fix6No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit
- -

How Not Helpful Comments Were Identified

+

Declined — correct but not adopted (27): All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong.

+

Not Helpful — incorrect (2):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
CommentRepo / PRWhy it was wrong
Mockito varargs matcher warningCommon #3171Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12.
"Missing @Override annotation"Broker #212The @Override annotation was already present — a hallucinated problem.
- - -

Results by Repository

- -
-
-
Broker
-
-
48.5%
-
16%
-
35%
-
-
293
-
-
-
Common
-
-
36.2%
-
18%
-
46%
-
-
188
-
-
-
MSAL
-
-
26%
-
25%
-
49%
-
-
76
-
-
-
- Helpful - Not Helpful - Unresolved - (number = total comments) -
- +

Unresolved (31): No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated.

+
+

Results by Repository

- - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
RepositoryCommentsResponse RateHelpfulDeclinedNot HelpfulUnresolvedPrecision
Broker6669.7%33 (50.0%)16 (24.2%)1 (1.5%)16 (24.2%)97.1%
Common5168.6%29 (56.9%)7 (13.7%)1 (2.0%)14 (27.5%)96.7%
MSAL1593.3%10 (66.7%)4 (26.7%)0 (0.0%)1 (6.7%)100%
- -

Broker has the highest response rate (56%) and the highest confirmed helpfulness (49%). In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved.

- - -

Results by Engineer

-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- +

Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository.

+

Coverage across the three repos:

- - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
RepositoryHuman PRsPRs Reviewed by CopilotCoverage
Common382361%
MSAL14750%
Broker352366%
- -
- Engagement drives value — and visibility. - Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
- - -

Response Behavior Deep Dive

- -

What happens when engineers reply? (239 comments)

-
-
60% helpful
-
40% not helpful
-
-

When engineers engage, the majority of Copilot feedback turns out useful. This is the strongest signal that review quality is decent — the bottleneck is adoption.

- -

What happens when engineers don't reply? (318 comments)

-
-
27% silently fixed
-
-
71% unresolved
-
- +
+

Results by Engineer

+

Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break.

- - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code silently applied (verified via diff)5015.7%Confirmed Helpful
Exact commented lines modified in subsequent commit72.2%Confirmed Helpful
Evidence of fix at nearby lines (re-audit)299.1%Confirmed Helpful
Merged without any subsequent commits12238.4%Unresolved
File modified but not at the commented lines4514.2%Unresolved
File modified but no line number to verify5015.7%Unresolved
File never modified after the comment92.8%Unresolved
Comment on stale/outdated code61.9%Confirmed Not Helpful
EngineerCommentsRepliedResponse RateHelpfulDeclinedNot HelpfulUnresolvedHelpfulnessPrecision
Engineer A77100%7000100%100%
Engineer B33100%3000100%100%
Engineer C11981.8%900281.8%100%
Engineer D6583.3%401166.7%80.0%
Engineer E302376.7%1770656.7%100%
Engineer F9555.6%520255.6%100%
Engineer G11654.5%601454.5%85.7%
Engineer H6350.0%300350.0%100%
Engineer I44100%220050.0%100%
Engineer J191578.9%890242.1%100%
Engineer K191578.9%870442.1%100%
Engineer L300%00030%n/a
Engineer M300%00030%n/a
Engineer N100%00010%n/a
- -

The single largest category is the 122 comments (38% of ignored) where the engineer merged the PR without pushing any additional commits after Copilot's review. These comments may have been valid, but we cannot tell because the engineer never evaluated them.

- - -

What Copilot Is Good At

- -

Catching real bugs:

+

Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.

+

Key observation — the Declined column changes how we read this table. Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the least value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a 100% precision: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket.

+
+

Response Behavior Deep Dive

+

Of the 132 total comments:

+
    +
  • 95 (72.0%) received a reply. Of those, 69.5% were helpful, 28.4% were declined (correct but not adopted), and only 2.1% were incorrect. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong.
    • +
  • 37 (28.0%) were not replied to. Of those, 16.2% were silently addressed (verified via diff) and the remaining 83.8% are unresolved — we cannot determine whether the comment was useful because the engineer never evaluated it.
    • +
+

What happens to comments engineers reply to

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Reply verdictCount% of repliedCounts against Copilot?
Helpful — accepted or fixed6669.5%No — positive
Declined — correct but not adopted2728.4%No — neutral
Not Helpful — incorrect22.1%Yes
+

The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot wrong only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently.

+

What happens to ignored comments

+ + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code / exact lines silently applied (verified via diff)616.2%Confirmed Helpful
No reply and no diff evidence either way3183.8%Unresolved
+

The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further.

+
+

What Copilot Is Good At

+

The most valuable helpful comments this period, with real examples from our PRs:

+

Catching real initialization-ordering bugs:

-

PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters.

-

Engineer reply: "You're right. Making the change."

+

PR #215 (Broker) — AriaInitializer: Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."

+

This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone.

- -

Stale documentation and naming inconsistencies:

+

Telemetry schema and contract discrepancies:

-

PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant USE_TEE_ONLY_FOR_TOKEN_BINDING after it was renamed. All four were silently fixed in the next commit.

+

PR #204 (Broker) — AccountChooser: Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +Engineer acknowledged and reconciled the schema.

- -

Dead code and unused imports:

+

Deprecated-API and compatibility issues:

-

PR #3040 (Common): Copilot spotted an unused local variable enabledSettingRaw. Verified as fixed via diff — the suggested replacement code appeared in the commit additions.

+

PR #3147 (Common): Copilot pointed out a newly-added overload shadowed a deprecated one and recommended @JvmOverloads / a default argument to preserve binary compatibility for Java callers. +Engineer reply: "Fixed. Thanks for pointing this out."

- -

CI/pipeline configuration issues:

+

Security and policy consistency:

-

PR #3038 (Common): Copilot warned that vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version.

+

PR #3156 (Common) — OWASP dependency check: Copilot identified an inconsistency in how a CVE suppression was applied across modules. +Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."

+

Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item.

- -

The @copilot apply workflow:

+

Behavior-vs-description correctness:

-

In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back with: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue.

+

PR #3165 (Common) — Authority validation: Copilot noted a URL comparison was matching on the full URL when only the host should matter. +Engineer reply: "Good point — updated the code to check host only."

- - -

What Copilot Struggles With

- -

Lacking domain context:

+
+

When Copilot Was Actually Wrong

+

Only two comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule:

+

Misjudging a library's behavior:

-

Copilot: "shared_device_id could be used for tracking across apps — consider hashing before emission."

-

Engineer: "The shared_device_id is a random UUID generated by one of the participating apps and is not PII."

+

PR #3171 (Common): Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."

+

Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive.

- -

Suggesting tests for trivial code:

+

Hallucinating a missing annotation:

-

Copilot: "New telemetry attributes lack test coverage..."

-

Engineer: "These are just telemetry related changes and adding unit tests will be overdo here."

+

PR #212 (Broker): Copilot claimed a method was missing an @Override annotation. +Engineer reply: "The override annotation is already there."

+

The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero.

- -

Misunderstanding APIs:

+
+

When Copilot Was Right but the Engineer Declined

+

These comments were not wrong — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction.

+

Deliberate notation choice (the flagship case):

-

Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."

-

Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online."

+

PR #201 (Broker): Copilot flagged the code comment ~1 per device as potentially misleading, since the tilde could be read as an exact value. +Engineer reply: "~ = approximate. i think this is ok."

+

A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure.

- -

Commenting on intentional design choices:

+

Subjective naming preference:

-

Copilot: "getPackageInfo() != null is redundant since it either returns PackageInfo or throws..."

-

Engineer: "This is fine. The verbosity makes the code clearer to understand."

+

PR #3151 (Common): Copilot suggested encoding the "prompt" semantics into a span name for clarity. +Engineer reply: "I think the current name is fine."

+

A judgment call on naming. Both names are defensible; the author preferred the existing one.

- - -

Key Takeaways

- - - - - - - - - - - -
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality — explicit engineer dismissals — the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49% helpful), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
8At ~5 comments per PR, the signal-to-noise question can't be settled without engagement. We confirmed ~2 useful comments per PR on average, but with 41% unresolved, the actual number could be higher.
- - -

How We Measured This

- +

Intentional verbosity for a new feature:

+
+

PR #223 (Broker): Copilot suggested not logging a full stack trace for an expected exception type. +Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."

+

The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here.

+
+

Context Copilot couldn't have known:

+
+

PR #3177 (Common): Copilot suggested removing an INFO-level log as noise. +Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."

+

The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff.

+
+
+

Trend Over Time

+

This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the Declined vs. Incorrect split and the Precision metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table.

- - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used the GitHub compare API to check if suggestion tokens appeared as diff additions, or if the exact line range was modified in subsequent commits.
4. AI-assisted classificationAll 133 ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong (e.g., typo fix + matching -1 line file change).
PeriodDaysCommentsComments/wkResponse RateHelpfulDismissed (combined)Unresolved
Jan 24 – Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 – Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 – Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 – Jul 153513226.472.0%51.5%22.0%26.5%
+

What changed this period:

+
    +
  • Response rate rose from 67.5% to 72.0%, driven partly by engineers going back to reply to previously-unanswered comments.
    • +
  • Trend-series helpfulness dipped slightly (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable.
    • +
  • The "Dismissed" number is misleading in isolation. This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that 20.5 of those 22.0 points are Declined (Copilot was right) and only 1.5 points are genuine errors. Copilot's precision this period is 97.3%. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong.
    • +
+

Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.

+

Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.

+
+

Key Takeaways

+
    +

  1. Copilot is almost never wrong — precision is 97.3%. Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74.

    +
    2. +

  2. "Declined" is not "wrong," and it's now counted fairly. 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent.

    +
    3. +

  3. Response rate climbed to 72%. Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments.

    +
    4. +

  4. 54.5% of comments led to a confirmed improvement. With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated.

    +
    5. +

  5. Every repository posts 96%+ precision. MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo.

    +
    6. +

  6. The Declined bucket concentrates in high-engagement engineers. Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem.

    +
    7. +

  7. Unresolved is the only real blind spot. 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report.

    +
    8. +

  8. The bottleneck is adoption and engagement, not accuracy. The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback.

    +
    9. +
+
+

What We'd Recommend

+

Based on this analysis, we see three opportunities:

+

1. Keep engaging — it's working. Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement.

+

2. Close the unresolved gap before merge. The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact.

+

3. Target the two real error patterns. The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining copilot-instructions.md — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero.

+
+

Methodology Notes

+
    +
  • Scope. Only inline code review comments from the Copilot user were counted. PR-level summary comments were excluded from the helpfulness analysis.
    • +
  • Bot exclusions. PRs authored by copilot-swe-agent (Copilot coding agent), dependabot[bot], and github-actions[bot] were excluded. Only PRs authored by human engineers were analyzed.
    • +
  • Three-way reply classification. Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is whether Copilot was factually/technically wrong, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers.
    • +
  • Precision. Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake.
    • +
  • Diff verification. For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as + lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed.
    • +
  • Conservative "Unresolved." Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful."
    • +
  • Account merging. Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings.
    • +
  • Data availability. Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at %TEMP%\copilot-review-analysis\ for independent verification.
    • +
+
+

Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.

-

Classification rules: A comment is "Confirmed Helpful" only with positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" only with positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage — including the final review round merged without evaluation — are classified as "Unresolved" rather than assumed unhelpful.

- -
-
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

-
- + \ No newline at end of file diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md index e0764b36..d4c6dda0 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md @@ -1,19 +1,21 @@ # Copilot Code Review Effectiveness Analysis -**Android Auth Platform | January 23 – March 23, 2026** +**Android Auth Platform | June 10 – July 15, 2026** --- ## Executive Summary -We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past two months. For each of the **557 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs. +We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the **132 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because *Copilot was wrong* or because *the engineer made a deliberate judgment call*. + +This run introduces a **new, fairer methodology**. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral **Declined** category and reserves **Not Helpful** exclusively for comments where Copilot was demonstrably *wrong*. **Key findings:** -- **57% of Copilot's comments received no response from engineers.** This is the single most important number in this report. The majority of AI review feedback is never even acknowledged. -- **Of comments that engineers engaged with, 60% were helpful** — a strong signal that the review quality itself is decent. -- **41% of all comments led to a confirmed code improvement.** But 41% are unresolved — comments that were ignored and where we lack evidence to judge either way. The true helpfulness rate could be significantly higher, but we can't confirm it because no one looked. -- **Engineers who reply to 75%+ of comments see 47-70% helpfulness.** Engineers who reply to <35% see 20-40%. The biggest lever for improving Copilot review value is not the AI — it's engineer engagement with the feedback. +- **72% of Copilot's comments now receive a response from engineers.** This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value. +- **Only 1.5% of all comments were genuinely wrong.** Across 132 comments, just **two** were confirmed as Copilot errors. When you compute Copilot's **precision** — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at **97.3%**. The AI is rarely incorrect. +- **20.5% of comments were correct-but-declined.** These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. **These do not count against Copilot's quality.** Treating them as failures, as the old methodology did, materially understated the tool's accuracy. +- **54.5% of all comments led to a confirmed code improvement.** With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy. --- @@ -21,17 +23,32 @@ We analyzed **every inline code review comment** left by GitHub Copilot on human This analysis went through five phases to ensure accuracy: -1. **Data collection.** We used the GitHub API to extract all 557 Copilot inline review comments from 163 human-authored PRs (excluding PRs authored by Copilot coding agent). We also recorded which comments received human replies and what those replies said. +1. **Data collection.** We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text. -2. **Reply-based classification.** For the 239 comments (43%) that received a human reply, we classified the reply as positive (e.g., "good catch", "fixed", "addressed"), negative (e.g., "won't fix", "not applicable", "by design"), or ambiguous. +2. **Diff-level verification.** For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. -3. **Diff-level verification.** For the 318 comments (57%) that received no reply, we checked whether the engineer acted on the feedback silently. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. +3. **AI-assisted reply classification (three-way).** For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts: + - **Helpful** — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot). + - **Declined** — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). **Copilot was not wrong** — it simply had no way to know the author's intent. + - **Not Helpful (incorrect)** — the engineer's reply demonstrates the comment was factually or technically *wrong* (a false positive, a hallucinated problem, or a misunderstanding of a library/API). -4. **AI-assisted reply classification.** All 133 comments with ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. For example, "this is just telemetry" was classified as dismissed, "@copilot apply changes" was classified as helpful (engineer delegated the fix back to Copilot), and "Added unit test for this" was classified as helpful (engineer acted on the suggestion). These classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. +4. **The fairness principle.** The dividing line between *Declined* and *Not Helpful* is **whether Copilot was wrong**, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is **Declined**, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as **Not Helpful**. -5. **Cross-validation.** All comments initially classified as "not helpful" were re-examined against the diff evidence. 18 were reclassified as helpful where the evidence was strong (e.g., a typo fix suggestion with a corresponding -1 line file change, or an unused import suggestion with a matching -2 line change). +5. **Cross-validation.** All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved. -The final dataset classifies each of the 557 comments as **confirmed helpful**, **confirmed not helpful**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. +The final dataset classifies each of the 132 comments as **confirmed helpful**, **declined (correct but not adopted)**, **not helpful (incorrect)**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. + +--- + +## A Note on Fairness: Why "Declined" Is Not "Wrong" + +The single most important change in this report is the recognition that **an engineer declining a comment is not evidence that the comment was bad.** + +Consider Broker PR #201. Copilot flagged a code comment that read `~1 per device` as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: *"~ = approximate. i think this is ok."* Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair. + +We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise. + +By separating these **Declined** cases (20.5% of all comments) from genuine **errors** (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually *wrong* in only 2 of 132 comments. --- @@ -39,11 +56,10 @@ The final dataset classifies each of the 557 comments as **confirmed helpful**, | Metric | Value | |--------|-------| -| Human PRs scanned | 163 | -| PRs that received Copilot review | 113 (69%) | -| Total inline review comments | 557 | -| PR-level summary comments | 205 | -| Average comments per reviewed PR | 4.9 | +| Human PRs scanned | 87 | +| PRs that received Copilot review | 53 (61%) | +| Total inline review comments | 132 | +| Average comments per reviewed PR | 2.5 | ### Engineer Response Rate @@ -51,220 +67,272 @@ Before looking at helpfulness, it's important to understand how engineers intera | Behavior | Count | Percentage | |----------|-------|------------| -| **Engineer replied** (any response — acceptance, dismissal, or discussion) | 239 | **42.9%** | -| **Engineer did not reply** | 318 | **57.1%** | +| **Engineer replied** (any response — acceptance, decline, or discussion) | 95 | **72.0%** | +| **Engineer did not reply** | 37 | **28.0%** | -More than half of all Copilot review comments receive no human response at all. This means the majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it. +At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a **100% response rate** across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted. ### Helpfulness Verdict -Each comment was classified into one of three categories: +Each comment was classified into one of four categories. Note the deliberate split between **Declined** (Copilot was right, the engineer chose otherwise) and **Not Helpful** (Copilot was wrong): | Verdict | Count | Percentage | Definition | |---------|-------|------------|------------| -| **Confirmed Helpful** | **230** | **41.3%** | The comment led to a code change — either the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | -| **Confirmed Not Helpful** | **101** | **18.1%** | The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design. We have positive evidence that the comment was *wrong*, not merely that it was *ignored*. | -| **Unresolved** | **226** | **40.6%** | The comment received no reply AND we could not confirm whether it was addressed. This includes cases where the engineer merged without any subsequent commits (the final review round was never evaluated), where the file was modified but not at the specific lines Copilot flagged, or where the comment had no line number making verification impossible. | +| **Confirmed Helpful** | **72** | **54.5%** | The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | +| **Declined (correct, not adopted)** | **27** | **20.5%** | The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. **This is neutral; Copilot was not wrong.** | +| **Not Helpful (incorrect)** | **2** | **1.5%** | The engineer demonstrated the comment was factually or technically *wrong* — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality. | +| **Unresolved** | **31** | **23.5%** | The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated. | + +### The Precision Metric + +Because *Declined* is neutral and *Unresolved* is unknown, the fairest single measure of Copilot's review quality is its **precision** — of the comments we can actually adjudicate as right or wrong, what fraction were right? + +> **Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%** -The **unresolved** category is the most important number in this report. These 226 comments — **41% of all feedback** — are not confirmed failures of the AI. They are comments where we simply lack evidence either way because no one engaged with them. Many may be perfectly valid feedback that was never read. If engineers had engaged with them (even just to dismiss), we would know. The true helpfulness rate likely falls somewhere between 41% (confirmed floor) and 82% (if all unresolved were helpful). +Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. **97.3% precision** means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy. ### How Each Category Breaks Down -**Confirmed Helpful (230):** +**Confirmed Helpful (72):** | Path | Count | Description | |------|-------|-------------| -| Engineer replied and acknowledged | 144 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "addressed", "added unit test") | -| Engineer silently applied the fix | 86 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | +| Engineer replied and acknowledged | 66 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …") | +| Engineer silently applied the fix | 6 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | -**Confirmed Not Helpful (101):** +**Declined — correct but not adopted (27):** All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong. -| Path | Count | Description | -|------|-------|-------------| -| Engineer replied and dismissed | 95 | Engineer explicitly explained why the comment was wrong, irrelevant, or by design (e.g., "won't fix", "this is fine", "Copilot is incorrect", "false positive") | -| Comment on code already changed | 6 | Comment was on stale/outdated code that had already been modified in a different commit | +**Not Helpful — incorrect (2):** -**Unresolved (226):** +| Comment | Repo / PR | Why it was wrong | +|---------|-----------|------------------| +| Mockito varargs matcher warning | Common #3171 | Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12. | +| "Missing `@Override` annotation" | Broker #212 | The `@Override` annotation was already present — a hallucinated problem. | -| Path | Count | Description | -|------|-------|-------------| -| Merged without any subsequent commits | 122 | No commits after Copilot's review — the engineer merged the PR without acting on the final review round. The comment may have been valid, but we cannot tell because the engineer never evaluated it. | -| File modified but not at the commented lines | 45 | The file was changed after the review, but the diff shows the changes were at different lines than what Copilot flagged. Possibly addressed via a different approach, or possibly coincidental. | -| File modified but no line number to verify | 50 | Copilot's comment had no line number metadata, and the file was modified. We cannot confirm whether the specific concern was addressed. | -| File never modified after the comment | 9 | The file was not touched in any commit after the review, and no reply was left. The comment may have been valid but was ignored. | +**Unresolved (31):** No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated. --- ## Results by Repository -| Repository | Comments | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | -|------------|----------|---------------|------|------|------| -| **Broker** | 293 | **56.0%** | 142 (48.5%) | 48 (16.4%) | 103 (35.2%) | -| **Common** | 188 | **29.8%** | 68 (36.2%) | 34 (18.1%) | 86 (45.7%) | -| **MSAL** | 76 | **25.0%** | 20 (26.3%) | 19 (25.0%) | 37 (48.7%) | +| Repository | Comments | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Precision | +|------------|----------|---------------|---------|----------|-------------|------------|-----------| +| **Broker** | 66 | **69.7%** | 33 (50.0%) | 16 (24.2%) | 1 (1.5%) | 16 (24.2%) | **97.1%** | +| **Common** | 51 | **68.6%** | 29 (56.9%) | 7 (13.7%) | 1 (2.0%) | 14 (27.5%) | **96.7%** | +| **MSAL** | 15 | **93.3%** | 10 (66.7%) | 4 (26.7%) | 0 (0.0%) | 1 (6.7%) | **100%** | -Broker has the highest response rate (56%) and correspondingly the highest confirmed helpfulness (49%). But even in Broker, 35% of comments are unresolved. In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved, meaning we cannot determine whether the AI's feedback was useful because engineers didn't evaluate it. +Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository. Coverage across the three repos: | Repository | Human PRs | PRs Reviewed by Copilot | Coverage | |------------|-----------|------------------------|----------| -| Common | 68 | 47 | 69% | -| MSAL | 31 | 19 | 61% | -| Broker | 64 | 47 | 73% | +| Common | 38 | 23 | 61% | +| MSAL | 14 | 7 | 50% | +| Broker | 35 | 23 | 66% | --- ## Results by Engineer -Each engineer has two GitHub accounts (a personal account for public repos and an EMU account for the private broker repo). These have been merged. Names are anonymized. +Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break. + +| Engineer | Comments | Replied | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Helpfulness | Precision | +|----------|----------|---------|---------------|---------|----------|-------------|------------|-------------|-----------| +| **Engineer A** | 7 | 7 | **100%** | 7 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer B** | 3 | 3 | **100%** | 3 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer C** | 11 | 9 | **81.8%** | 9 | 0 | 0 | 2 | **81.8%** | 100% | +| **Engineer D** | 6 | 5 | **83.3%** | 4 | 0 | 1 | 1 | **66.7%** | 80.0% | +| **Engineer E** | 30 | 23 | **76.7%** | 17 | 7 | 0 | 6 | **56.7%** | 100% | +| **Engineer F** | 9 | 5 | **55.6%** | 5 | 2 | 0 | 2 | **55.6%** | 100% | +| **Engineer G** | 11 | 6 | **54.5%** | 6 | 0 | 1 | 4 | **54.5%** | 85.7% | +| **Engineer H** | 6 | 3 | **50.0%** | 3 | 0 | 0 | 3 | **50.0%** | 100% | +| **Engineer I** | 4 | 4 | **100%** | 2 | 2 | 0 | 0 | **50.0%** | 100% | +| **Engineer J** | 19 | 15 | **78.9%** | 8 | 9 | 0 | 2 | **42.1%** | 100% | +| **Engineer K** | 19 | 15 | **78.9%** | 8 | 7 | 0 | 4 | **42.1%** | 100% | +| **Engineer L** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer M** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer N** | 1 | 0 | **0%** | 0 | 0 | 0 | 1 | **0%** | n/a | + +*Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.* + +**Key observation — the Declined column changes how we read this table.** Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the *least* value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a **100% precision**: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket. -| Engineer | Comments | Replied | Ignored | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | Helpfulness | -|----------|----------|---------|---------|---------------|------|------|------|-------------| -| **Engineer A** | 20 | 20 | 0 | **100%** | 14 | 6 | 0 | **70.0%** | -| **Engineer B** | 83 | 75 | 8 | **90.4%** | 57 | 26 | 0 | **68.7%** | -| **Engineer C** | 15 | 4 | 11 | **26.7%** | 8 | 2 | 5 | **53.3%** | -| **Engineer D** | 40 | 30 | 10 | **75.0%** | 19 | 14 | 7 | **47.5%** | -| **Engineer E** | 110 | 37 | 73 | **33.6%** | 44 | 18 | 48 | **40.0%** | -| **Engineer F** | 99 | 20 | 79 | **20.2%** | 36 | 11 | 52 | **36.4%** | -| **Engineer G** | 63 | 22 | 41 | **34.9%** | 20 | 12 | 31 | **31.7%** | -| **Engineer H** | 100 | 24 | 76 | **24.0%** | 27 | 12 | 61 | **27.0%** | -| **Engineer I** | 24 | 6 | 18 | **25.0%** | 5 | 0 | 19 | **20.8%** | -| **Engineer J** | 3 | 1 | 2 | **33.3%** | 0 | 0 | 3 | **0%** | +--- -*Helpfulness = Confirmed Helpful / Total Comments. Response Rate = Replied / Total Comments. Unresolved = comments with no reply and no definitive diff evidence either way.* +## Response Behavior Deep Dive -**Key observation:** There is a strong correlation between response rate and helpfulness. Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness (70% and 69%) — and crucially, **zero unresolved comments**. When engineers engage, we know exactly what's helpful and what's not. Engineers with low response rates (Engineer F at 20%, Engineer H at 24%) have massive unresolved buckets (52 and 61 comments respectively) — over half their comments go into a black hole where we can't tell if the AI was right or wrong. +Of the 132 total comments: ---- +- **95 (72.0%) received a reply.** Of those, **69.5% were helpful**, **28.4% were declined** (correct but not adopted), and only **2.1% were incorrect**. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong. +- **37 (28.0%) were not replied to.** Of those, **16.2% were silently addressed** (verified via diff) and the remaining **83.8% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. -## Response Behavior Deep Dive +### What happens to comments engineers reply to -Of the 557 total comments: +| Reply verdict | Count | % of replied | Counts against Copilot? | +|---------------|-------|-------------|-------------------------| +| Helpful — accepted or fixed | 66 | 69.5% | No — positive | +| Declined — correct but not adopted | 27 | 28.4% | **No — neutral** | +| Not Helpful — incorrect | 2 | 2.1% | Yes | -- **239 (42.9%) received a reply.** Of those, **60.3% were helpful** and 39.7% were not helpful. When engineers engage, the majority of Copilot feedback turns out to be useful. -- **318 (57.1%) were ignored.** Of those, **27.0% were silently addressed** (verified via diff), **1.9% were on stale code** (confirmed not helpful), and the remaining **71.1% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. +The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot *wrong* only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently. ### What happens to ignored comments | What happened | Count | % of ignored | Verdict | |---------------|-------|-------------|---------| -| Suggestion code silently applied (verified via diff) | 50 | 15.7% | Confirmed Helpful | -| Exact commented lines modified in subsequent commit | 7 | 2.2% | Confirmed Helpful | -| Re-audit: evidence of fix at nearby lines | 29 | 9.1% | Confirmed Helpful | -| Merged without any subsequent commits | 122 | 38.4% | **Unresolved** | -| File never modified after the comment | 9 | 2.8% | **Unresolved** | -| Comment on stale/outdated code | 6 | 1.9% | Confirmed Not Helpful | -| File modified but not at the commented lines | 45 | 14.2% | **Unresolved** | -| File modified but no line number to verify | 50 | 15.7% | **Unresolved** | +| Suggestion code / exact lines silently applied (verified via diff) | 6 | 16.2% | Confirmed Helpful | +| No reply and no diff evidence either way | 31 | 83.8% | **Unresolved** | -The single largest category is the **122 comments (38.4% of ignored)** where the engineer merged the PR without pushing any additional commits after Copilot's review. These represent the final review round being skipped entirely — the feedback had zero chance of impact regardless of its quality. +The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further. --- ## What Copilot Is Good At -The most common categories of helpful comments, with real examples from our PRs: +The most valuable helpful comments this period, with real examples from our PRs: -**Catching real bugs:** -> *PR #3050 (Common):* Copilot flagged that `"$it"` string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters. -> *Engineer reply: "You're right. Making the change."* +**Catching real initialization-ordering bugs:** +> *PR #215 (Broker) — `AriaInitializer`:* Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +> *Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."* +> +> This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone. -**Stale documentation and naming inconsistencies:** -> *PR #64 (Broker):* Copilot identified four locations where KDoc still referenced the old flight constant `USE_TEE_ONLY_FOR_TOKEN_BINDING` after it was renamed to `USE_TEE_ONLY_FOR_HARDWARE_BOUND_KEYS`. All four were silently fixed in the next commit. +**Telemetry schema and contract discrepancies:** +> *PR #204 (Broker) — `AccountChooser`:* Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +> *Engineer acknowledged and reconciled the schema.* -**Dead code and unused imports:** -> *PR #3040 (Common):* Copilot spotted an unused local variable `enabledSettingRaw` that was assigned but never read. Verified as fixed via diff analysis — the suggested replacement code appeared in the commit additions. +**Deprecated-API and compatibility issues:** +> *PR #3147 (Common):* Copilot pointed out a newly-added overload shadowed a deprecated one and recommended `@JvmOverloads` / a default argument to preserve binary compatibility for Java callers. +> *Engineer reply: "Fixed. Thanks for pointing this out."* -**CI/pipeline configuration issues:** -> *PR #3038 (Common):* Copilot warned that using `vmImage: 'windows-latest'` makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. +**Security and policy consistency:** +> *PR #3156 (Common) — OWASP dependency check:* Copilot identified an inconsistency in how a CVE suppression was applied across modules. +> *Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."* +> +> Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item. -**The `@copilot apply` workflow:** -> In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back to Copilot with: `@copilot open a new pull request to apply changes based on [this feedback]`. This is an efficient pattern where AI identifies and fixes the issue end-to-end. +**Behavior-vs-description correctness:** +> *PR #3165 (Common) — Authority validation:* Copilot noted a URL comparison was matching on the full URL when only the host should matter. +> *Engineer reply: "Good point — updated the code to check host only."* --- -## What Copilot Struggles With +## When Copilot Was Actually Wrong -The most common categories of unhelpful comments, with real examples: +Only **two** comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule: -**Lacking domain context:** -> *Copilot:* "`shared_device_id` could be used for tracking across apps — consider hashing before emission." -> *Engineer:* "The shared_device_id is a random UUID generated by one of the participating apps and is not PII." +**Misjudging a library's behavior:** +> *PR #3171 (Common):* Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +> *Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."* > -> Copilot applied a general security heuristic without understanding that this particular identifier is already random and non-linkable. +> Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive. -**Suggesting tests for trivial code:** -> *Copilot:* "New telemetry attributes lack test coverage..." -> *Engineer:* "These are just telemetry related changes and adding unit tests will be overdo here." +**Hallucinating a missing annotation:** +> *PR #212 (Broker):* Copilot claimed a method was missing an `@Override` annotation. +> *Engineer reply: "The override annotation is already there."* > -> This was a recurring theme — Copilot frequently requests tests for logging/telemetry code that the team considers low-risk. +> The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero. -**Misunderstanding APIs:** -> *Copilot:* "`00000003-0000-0ff1-ce00-000000000000` is the resource ID for Microsoft Graph, not SharePoint Online." -> *Engineer:* "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." +--- + +## When Copilot Was Right but the Engineer Declined + +These comments were **not wrong** — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction. + +**Deliberate notation choice (the flagship case):** +> *PR #201 (Broker):* Copilot flagged the code comment `~1 per device` as potentially misleading, since the tilde could be read as an exact value. +> *Engineer reply: "~ = approximate. i think this is ok."* +> +> A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure. + +**Subjective naming preference:** +> *PR #3151 (Common):* Copilot suggested encoding the "prompt" semantics into a span name for clarity. +> *Engineer reply: "I think the current name is fine."* > -> Copilot was factually wrong about a well-known Microsoft service resource ID. +> A judgment call on naming. Both names are defensible; the author preferred the existing one. -**Commenting on intentional design choices:** -> *Copilot:* "`getPackageInfo() != null` is redundant since it either returns PackageInfo or throws..." -> *Engineer:* "This is fine. The verbosity makes the code clearer to understand." +**Intentional verbosity for a new feature:** +> *PR #223 (Broker):* Copilot suggested not logging a full stack trace for an expected exception type. +> *Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."* +> +> The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here. -**Over-engineering suggestions:** -> *Copilot:* "Use a bounded min-heap of size `PRT_ARTIFACT_LIMIT` to reduce overhead..." -> *Engineer:* (The list can only ever contain 3 items — there are only 3 broker apps on Android.) +**Context Copilot couldn't have known:** +> *PR #3177 (Common):* Copilot suggested removing an INFO-level log as noise. +> *Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."* +> +> The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff. --- -## Most Reviewed Files +## Trend Over Time + +This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the **Declined vs. Incorrect** split and the **Precision** metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table. + +| Period | Days | Comments | Comments/wk | Response Rate | Helpful | Dismissed (combined) | Unresolved | +|--------|------|----------|-------------|---------------|---------|----------------------|------------| +| Jan 24 – Mar 25 | 60 | 570 | 66.3 | 44.4% | 38.6% | 17.4% | 44.0% | +| Mar 24 – Apr 19 | 26 | 170 | 45.9 | 78.8% | 62.9% | 21.2% | 15.9% | +| Apr 20 – Jun 9 | 50 | 157 | 22.1 | 67.5% | 52.9% | 16.6% | 30.6% | +| **Jun 10 – Jul 15** | **35** | **132** | **26.4** | **72.0%** | **51.5%** | **22.0%** | **26.5%** | -| Rank | File | Comments | -|------|------|----------| -| 1 | `.github/workflows/copilot-issue-response.yml` | 27 | -| 2 | `broker4j/.../AttributeName.java` | 19 | -| 3 | `.github/workflows/copilot-ci-feedback.md` | 16 | -| 4 | `common/.../AuthorizationFragment.java` | 15 | -| 5 | `broker4j/.../MultipleWorkplaceJoinDataStore.java` | 11 | -| 6 | `broker4j/.../AbstractBrokerController.java` | 11 | -| 7 | `common/.../AzureActiveDirectoryWebViewClient.java` | 11 | -| 8 | `AADAuthenticator/.../BrowserSsoProvider.kt` | 10 | -| 9 | `broker4j/.../BrokerFlight.java` | 9 | -| 10 | `broker4j/.../DeviceRegistrationRequestHandler.java` | 9 | +**What changed this period:** -CI/workflow files and large Java classes with many change touchpoints attract the highest volume of comments. `AttributeName.java` appears frequently because many PRs add telemetry attributes. +- **Response rate rose from 67.5% to 72.0%**, driven partly by engineers going back to reply to previously-unanswered comments. +- **Trend-series helpfulness dipped slightly** (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable. +- **The "Dismissed" number is misleading in isolation.** This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that **20.5 of those 22.0 points are Declined (Copilot was right)** and only **1.5 points are genuine errors**. Copilot's **precision this period is 97.3%**. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong. + +*Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.* + +*Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.* --- ## Key Takeaways -1. **57% of Copilot review comments receive no response from engineers.** This is the most significant finding. The majority of AI review feedback is never acknowledged. Some of it is silently acted on (27% of ignored comments show diff evidence of a fix), but the vast majority — 71% of ignored comments — are unresolved. We simply don't know if they were useful because no one evaluated them. +1. **Copilot is almost never wrong — precision is 97.3%.** Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74. + +2. **"Declined" is not "wrong," and it's now counted fairly.** 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent. + +3. **Response rate climbed to 72%.** Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments. + +4. **54.5% of comments led to a confirmed improvement.** With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated. -2. **41% of all comments led to a confirmed code improvement.** But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement. +5. **Every repository posts 96%+ precision.** MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo. -3. **When engineers engage, 60% of comments are helpful.** Of the 239 comments that received a reply, 144 (60%) led to acknowledged improvements. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy. +6. **The Declined bucket concentrates in high-engagement engineers.** Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem. -4. **Only 18% of comments are confirmed not helpful.** When we restrict "not helpful" to comments where the engineer explicitly dismissed the feedback — the only cases where we have positive evidence of low quality — the rate is surprisingly low. +7. **Unresolved is the only real blind spot.** 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report. -5. **Engagement is the strongest predictor of value.** Engineers who reply to 75%+ of comments see 47-70% confirmed helpfulness with zero unresolved comments. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets (50-60% of their comments). The tool works better when engineers work with it. +8. **The bottleneck is adoption and engagement, not accuracy.** The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback. -6. **38% of ignored comments are on the final commit before merge.** These 122 comments represent the last review round being skipped entirely — the engineer merged without pushing any further changes. These comments may have been perfectly valid, but the feedback had zero chance of impact. +--- + +## What We'd Recommend + +Based on this analysis, we see three opportunities: -7. **Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%).** This correlates with response rate: Broker engineers reply to 56% of comments, while Common (30%) and MSAL (25%) reply far less. +**1. Keep engaging — it's working.** Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement. -8. **At ~5 comments per PR, the signal-to-noise conversation can't be settled without engagement.** We confirmed ~2 useful comments per PR on average, but with 41% of comments unresolved, the actual number could be higher. The only way to know is for engineers to evaluate the feedback. +**2. Close the unresolved gap before merge.** The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact. + +**3. Target the two real error patterns.** The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining `copilot-instructions.md` — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero. --- ## Methodology Notes -- **Scope.** Only inline code review comments from the `Copilot` and `copilot-pull-request-reviewer[bot]` users were counted. PR-level summary comments (205 total) were excluded from the helpfulness analysis. -- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent) were excluded. Only PRs authored by human engineers were analyzed. -- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` (addition) lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped with the comment's target line range (±5 line tolerance). This is a conservative approach — some fixes that refactored code differently than suggested may be missed. -- **AI-assisted reply classification.** Every comment that could not be definitively classified by automated methods was individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. These AI classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. -- **Conservative approach.** We only classify a comment as "Confirmed Helpful" when there is positive evidence: an explicit engineer acknowledgment, or verified code changes at the exact lines/tokens suggested. We only classify as "Confirmed Not Helpful" when there is positive evidence that the comment was *wrong or irrelevant*: an explicit engineer dismissal with a stated reason. Comments where the engineer simply did not engage — including the final review round that was merged without evaluation, files that were never modified, and files modified at different lines — are classified as "Unresolved" rather than assumed to be unhelpful. -- **Account merging.** Engineers with separate public GitHub accounts and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. -- **Data availability.** Raw data for all 557 comments (including comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. +- **Scope.** Only inline code review comments from the `Copilot` user were counted. PR-level summary comments were excluded from the helpfulness analysis. +- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent), `dependabot[bot]`, and `github-actions[bot]` were excluded. Only PRs authored by human engineers were analyzed. +- **Three-way reply classification.** Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is **whether Copilot was factually/technically wrong**, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers. +- **Precision.** Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake. +- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed. +- **Conservative "Unresolved."** Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful." +- **Account merging.** Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. +- **Data availability.** Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. --- -*Analysis conducted March 23-24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.* +*Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.* + diff --git a/.github/skills/copilot-review-analyst/references/classification-rules.md b/.github/skills/copilot-review-analyst/references/classification-rules.md index 7685b431..644af53e 100644 --- a/.github/skills/copilot-review-analyst/references/classification-rules.md +++ b/.github/skills/copilot-review-analyst/references/classification-rules.md @@ -4,10 +4,25 @@ Guide for the AI agent performing Phase 3 reply classification. Read each replie ## Phase 3: Classifying Replied Comments -For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of: +For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of **three** verdicts: - **`helpful`** — The engineer's reply indicates Copilot's feedback led to (or will lead to) a code improvement -- **`not-helpful`** — The engineer's reply indicates Copilot's feedback was wrong, irrelevant, or not actionable +- **`not-helpful`** — Copilot's feedback was **factually wrong** (a false positive, hallucination, or a demonstrable error about the code/library/framework). This is the only bucket that counts *against* Copilot's review quality. +- **`declined`** — Copilot's feedback was **correct or reasonable**, but the engineer **intentionally chose not to act** on it for a by-design, subjective, or contextual reason (or the code became moot). This is **neutral** — it is *not* counted against Copilot, because Copilot had no way to know the author's intent beforehand. + +### Why `declined` exists (the fairness principle) + +A dismissal is **not** evidence that Copilot was wrong. Copilot reviews a diff without the author's design intent, offline discussions, or downstream context. When an engineer replies "intentional" or "this is fine as-is," Copilot's observation was often perfectly valid — the engineer simply made a judgment call. Counting these against Copilot punishes it for not being a mind-reader and understates real review quality. + +So we split the old "not-helpful" bucket into two: +- **`not-helpful` (incorrect)** — Copilot was actually *wrong*. Fair to count against it. +- **`declined` (by-design / subjective / moot)** — Copilot was *right or reasonable*, engineer declined. Neutral. + +The **Copilot precision** metric measures correctness only where it is evaluable: + +> **precision = helpful ÷ (helpful + not-helpful-incorrect)** + +`declined` and `unresolved` are excluded from both numerator and denominator. Precision answers: *"When Copilot spoke and we could judge correctness, how often was it actually right?"* ### What counts as Helpful @@ -18,26 +33,88 @@ For every comment where `HasReply = true`, read the `CommentBody` (what Copilot - **Linked a commit**: Reply contains a commit SHA or link showing they applied a fix - **Acknowledged for future**: "this can be considered in another PR" (acknowledges the issue is valid) -### What counts as Not Helpful +### What counts as Not Helpful (Copilot incorrect) + +Reserve this bucket for cases where the engineer's reply demonstrates Copilot was **factually wrong**: -- **Explicit dismissal**: "won't fix", "by design", "intentional", "not applicable", "false positive", "not relevant" -- **Copilot was wrong**: "incorrect", "Copilot is wrong", "hallucinating", "not accurate", "misunderstanding" -- **Already handled**: "already done", "already handled", "this is fine" -- **Explained away**: Engineer explains why the suggestion doesn't apply — "this is just telemetry", "we consciously chose this", "legacy code", "overdo", "only used in X context", "can't happen" -- **Dismissed or outdated**: "outdated", "dismissed", "out of scope" +- **Disproven claim**: Engineer verified the opposite — "Verified this isn't reproducible on Mockito 5.11.0; the suite passes 12/12" (Copilot was wrong about library behavior) +- **Hallucination**: Copilot flagged something that isn't there — "The `@Override` annotation is already there" (Copilot claimed it was missing) +- **Wrong about the code/framework/version**: Copilot's premise is objectively false for this codebase +- **False positive with proof**: Engineer shows the flagged issue cannot occur + +If the engineer *demonstrates* Copilot got a fact wrong → `not-helpful`. + +### What counts as Declined (neutral — Copilot correct/reasonable, engineer declined) + +- **By-design**: "intentional", "by design", "this is intentional", "we consciously chose this", "keeping as-is" +- **Subjective / nitpick**: "too nitpicky", "the current name is fine", "minor, no-fix", "imo this reads fine" +- **Contextual reason to keep**: "we need this log to catch a regression in cobo/cope", "for a new feature we want the full stack trace" +- **Acceptable risk**: "this should be okay since it's in a log file" +- **Moot / obviated**: "not relevant now that we removed this change", "outdated" (the code changed so the comment no longer applies — Copilot wasn't *wrong*, just stale) +- **Deprecated target**: "this config is deprecated, not fixing" +- **Accurate but declined**: engineer explicitly says the comment *was* accurate but chose not to act ("this is outdated, but was accurate") + +The test: *Did the engineer demonstrate Copilot was wrong (→ not-helpful), or did they acknowledge/ignore a valid-but-unwanted point (→ declined)?* When in doubt between `not-helpful` and `declined`, prefer **`declined`** unless there is positive evidence Copilot was factually incorrect. ### Edge Cases - **Mixed signals** (both positive and negative in same reply): Read the full reply to determine the engineer's overall intent. Don't rely on individual words — understand the sentence. -- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **not-helpful** if they're brushing it off. -- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer whether the engineer is acknowledging or dismissing. When genuinely unclear, lean toward **not-helpful** (conservative). +- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **declined** if they're setting it aside as by-design/out-of-scope (not because Copilot was wrong). +- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer intent. If it reads as acknowledgment → helpful. If it reads as a soft dismissal of a valid point → declined. Only use **not-helpful** when there is evidence Copilot was factually wrong. +- **"Outdated"**: Treat as **declined** (the code moved on; not a Copilot correctness failure) unless the engineer indicates Copilot's underlying claim was itself wrong. **Before scoring, apply the Force-Push Confound check below** — an "outdated" marker on a rewritten snapshot can hide a comment that was *correct* when Copilot reviewed it, and must never be recorded as `not-helpful`. ### Important: Read the Full Reply Do NOT use simple keyword matching. Read the engineer's full reply in context. For example: - "This won't fix the actual issue we're seeing" — This is NOT a "won't fix" dismissal; the engineer is discussing a different topic -- "Thanks but this is intentional" — Despite "thanks", this is a dismissal +- "Thanks but this is intentional" — Despite "thanks", this is a **declined** (by-design), not a Copilot error - "I disagree with this specific suggestion but good catch on the typo above" — Mixed; classify based on the primary concern +- "~ = approximate, I think this is ok" — Copilot's readability nit was reasonable; engineer made a judgment call → **declined**, not not-helpful + +## The Force-Push Confound (mandatory validation before scoring) + +**A "correct" Copilot comment can look like a false positive when the engineer rewrites history.** If an engineer rebases or force-pushes *after* Copilot reviews — a common habit right after opening a PR — the commit Copilot reviewed is orphaned. GitHub then marks the comment **"outdated"** and the visible code no longer matches what Copilot saw, so a comment that was *accurate at the reviewed snapshot* reads as stale or even hallucinated. Left unchecked, this silently inflates the `declined`/`unresolved` buckets and, in the worst case, scores a **correct** comment as `not-helpful` — understating precision. + +> This was not hypothetical: it caused the single decisive misclassification in the Jul 2026 audit (broker #212's `@Override` comment was mislabeled a hallucination; the `@Override` was genuinely absent at the reviewed commit `bcf360ea`, which the engineer force-pushed away ~2 min later). Precision moved **up** once corrected. + +### Detection — three immutable fields + +Phase 1 (`analyze.ps1`) now records these per comment in `raw_results.json`: + +- **`OriginalCommitId`** — `original_commit_id`, the exact commit Copilot reviewed. Immutable; survives force-push. +- **`ReviewedCommitRewritten`** — `true` when `OriginalCommitId` is **absent** from the PR's current commit list (`/pulls/{n}/commits`), i.e. the reviewed snapshot was rewritten. This is the force-push fingerprint. +- **`DiffHunk`** — `diff_hunk`, a frozen copy of the exact lines Copilot saw. Ground truth for what Copilot was actually looking at. + +`precise.ps1` carries `ReviewedCommitRewritten` forward and flags any `not-applied`/`file-not-changed` verdict that sits on a rewritten snapshot. + +### The rule + +For **every** comment classified `declined`, `not-helpful`, `unresolved`, or a no-reply `not-applied`/`file-not-changed` verdict, **first check `ReviewedCommitRewritten`**: + +1. If `false` → classify normally (no confound). +2. If `true` → the dismissal is **unsafe to credit as-is**. Re-derive Copilot's claim from the immutable `DiffHunk` (and, if needed, the reviewed file via `contents?ref=`) and compare it to the **merged** state: + - Copilot's claim was **true at the reviewed snapshot** and the merged code already satisfies it → the fix was effectively adopted by the rewrite → **helpful** (never `not-helpful`, never `declined`). + - Copilot's claim was **true at the reviewed snapshot** and the engineer explicitly declined it on the merits ("intentional", "okay in a log file") → **declined** (the rewrite was incidental). + - Copilot's claim was **false even at the reviewed snapshot** → `not-helpful` (a genuine error). + +**Never assign `not-helpful` to a comment on a rewritten snapshot without confirming Copilot was wrong at `OriginalCommitId`.** A stale marker is not evidence of a Copilot error. + +### Precision is computed only after this validation + +Because a mis-scored force-push case lands directly in the precision denominator, **precision must be recomputed only after the force-push validation pass has run** over all `declined`/`not-helpful`/`unresolved` comments. Report the count of `ReviewedCommitRewritten = true` comments as a measurement-integrity note. + +### Reproducible check (per comment) + +```bash +# 1. The commit Copilot reviewed (immutable): +gh api repos//pulls/comments/ --jq '.original_commit_id' +# 2. Is it still in the PR's commit list? (empty output => rewritten away => confound) +gh api repos//pulls//commits --paginate --jq '.[].sha' | grep +# 3. Ground truth Copilot saw: +gh api repos//pulls/comments/ --jq '.diff_hunk' +# 4. (optional) The reviewed file as Copilot saw it: +gh api "repos//contents/?ref=" --jq '.content' | base64 -d +``` ## Phase 3 Output Format @@ -51,11 +128,11 @@ Map of comment ID to verdict for every replied comment: { "1234567890": "helpful", "1234567891": "not-helpful", - "1234567892": "helpful" + "1234567892": "declined" } ``` -Keys are comment IDs (as strings). Values are `"helpful"` or `"not-helpful"`. +Keys are comment IDs (as strings). Values are `"helpful"`, `"not-helpful"` (Copilot incorrect), or `"declined"` (Copilot correct/reasonable but engineer declined). ### `reaudit-flips.json` @@ -81,7 +158,9 @@ These are handled by `precise.ps1` and `final-classification.ps1` automatically: | `suggestion-applied`, `suggestion-likely-applied`, `exact-lines-modified` | **Helpful** | | `lines-modified-different-fix` | **Helpful** | | `file-changed-elsewhere`, `file-changed-no-line-info` | **Not Helpful** (unless in re-audit flips) | -| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** | +| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** (unless in re-audit flips) | + +> **Force-push caveat:** any `not-applied` / `file-not-changed` row with `ReviewedCommitRewritten = true` is UNSAFE to score as Not Helpful — the reviewed snapshot was rewritten, so the fix may live in the orphaned commit. Validate against the immutable `DiffHunk` (see "The Force-Push Confound") before finalizing; flip to **Helpful** if Copilot's claim was already satisfied by the merged code. ## Account Mapping diff --git a/.github/skills/copilot-review-analyst/references/report-formatting.md b/.github/skills/copilot-review-analyst/references/report-formatting.md index b74db5b4..00961c2f 100644 --- a/.github/skills/copilot-review-analyst/references/report-formatting.md +++ b/.github/skills/copilot-review-analyst/references/report-formatting.md @@ -11,18 +11,20 @@ Generate both formats. Templates are in `assets/` within this skill folder. | # | Section | Content | Depth | |---|---------|---------|-------| | 1 | **Background** | Team context, what repos are covered, what was enabled | 2-3 prose paragraphs | -| 2 | **At a Glance** | 4 summary cards (no-response %, helpful %, not-helpful %, unresolved %) + callout about adoption | Cards + 1 detailed callout box | -| 3 | **Overall Results** | Response rate bar, helpfulness verdict bar, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + 2 breakdown tables | -| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/not/unresolved) | Bar per repo + data table + 1 interpretive paragraph | +| 2 | **At a Glance** | 5 summary cards (helpful %, declined %, incorrect %, unresolved %, precision) + callout explaining the fairness correction | Cards + 1 detailed callout box | +| 3 | **Overall Results** | Response rate bar, four-way verdict bar, precision formula, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + breakdown tables | +| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/declined/incorrect/unresolved/precision) | Bar per repo + data table + 1 interpretive paragraph | | 5 | **Results by Engineer** | Table with colored columns (anonymize names for org-wide sharing) | Full table + blue callout box highlighting the engagement-value correlation | | 6 | **Response Behavior Deep Dive** | What happens to ignored comments (silently applied, merged without commits, etc.) | Summary stats + detailed breakdown table + interpretive paragraph | | 7 | **What Copilot Is Good At** | 4-5 real examples with PR references and engineer quotes | Each example: category header + full Copilot comment text (not truncated) + engineer reply + 1-2 sentence explanation | -| 8 | **What Copilot Struggles With** | 4-5 real examples showing false positives, domain gaps | Same format as above — full quotes + explanatory context | +| 8 | **When Copilot Was Wrong / When Feedback Was Declined** | Genuine errors and correct-but-declined examples in separate subsections | Full quotes + explanatory context; never present a deliberate decline as a Copilot error | | 9 | **Most Reviewed Files** | Top 10 files by comment count | Table + 1 interpretive paragraph | | 10 | **Key Takeaways** | 7-8 numbered findings | Each finding: bold stat + explanatory sentence | | 11 | **Recommendations** | 3 actionable next steps | Each recommendation: 1 full prose paragraph (not a bullet point) with reasoning | | 12 | **Methodology Notes** | How data was collected, classified, and validated | 5-6 bullet points with sufficient detail for reproducibility | +**Headline framing (all formats).** The hero band and the first Key Takeaway lead with **Helpful & adopted %** (adoption), *not* precision. Adoption answers "did engineers act on the feedback?" — the question leadership cares about — and precision is presented as the supporting second number. Leading with precision (which is near-100% by construction once Declined/Unresolved are excluded) overstates the story and buries the adoption signal. + ## Trend Section (Section 2.5 — between At a Glance and Overall Results) **Only generated when `history.json` has ≥2 entries.** Skip entirely on the first run. @@ -35,11 +37,12 @@ Load `~/.copilot-review-analysis/history.json`. Entries are sorted newest-first. Since periods may have different lengths (e.g., 60 days vs 14 days): -1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, not-helpful %, unresolved %, and replied-helpful rate are directly comparable across any period length. +1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, combined dismissal %, unresolved %, and replied-helpful rate are directly comparable across any period length. For snapshots created after the three-way methodology change, also show Declined %, Incorrect %, and Precision. 2. **Show counts as context only.** Display total comments alongside `comments/week` for normalized volume comparison. Do NOT compute count deltas like "comments dropped from 570 to 85" — this is misleading when periods differ. 3. **Show period duration prominently.** Every trend row must include the date range and duration (e.g., "Jan 24–Mar 25 (60d)"). 4. **Use "pp" (percentage points) for deltas.** "↑ +7.6pp" not "↑ +7.6%". The delta is the arithmetic difference between two percentages. 5. **Color-code deltas.** Green (↑) for improvements (response rate up, helpful up, not-helpful down, unresolved down). Red (↓) for regressions. +6. **Preserve trend continuity.** Historical snapshots created before the `declined` verdict only have a combined dismissal rate. Label that column **Dismissed (combined)** and explain that the current period can be split into Declined (neutral) and Incorrect (counts against Copilot). Do not retroactively infer the split for old periods. ### Markdown Format (2 runs — current vs previous) @@ -98,10 +101,19 @@ From `final_classification.json`: # Overall $total = $data.Count $helpful = ($data | Where-Object { $_.Verdict -eq "helpful" }).Count -$notHelpful = ($data | Where-Object { $_.Verdict -eq "not-helpful" }).Count -$unresolved = $total - $helpful - $notHelpful +$declined = ($data | Where-Object { $_.Verdict -eq "declined" }).Count +$incorrect = ($data | Where-Object { + $_.Replied -eq $true -and $_.Verdict -eq "not-helpful" +}).Count +$unresolved = ($data | Where-Object { + $_.Replied -eq $false -and $_.Verdict -eq "not-helpful" +}).Count $replied = ($data | Where-Object { $_.Replied -eq $true }).Count $responseRate = [math]::Round(($replied / $total) * 100, 1) +$precisionDenominator = $helpful + $incorrect +$precision = if ($precisionDenominator -gt 0) { + [math]::Round(($helpful / $precisionDenominator) * 100, 1) +} else { 0 } # Per-repo $repoStats = $data | Group-Object Repo | ForEach-Object { ... } @@ -112,10 +124,12 @@ $engStats = $data | Group-Object Engineer | ForEach-Object { ... } ## Outlook HTML Formatting Rules +> **Canonical design:** `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html` is the current **v3 newsletter** template — read it in full before generating. The primitives below (table layout, inline styles, `bgcolor`+inline pairing, bars-not-SVG) still hold; the load-bearing v3 techniques (single-row chips, edge-to-edge frame, newsletter masthead, `mso-line-height-rule`) are documented under **v3 Newsletter Rendering** below. The asset is written entirely single-quoted so it stays escaping-free (`"`/`\` count = 0) for a trivial Graph push. + Outlook strips most modern CSS. Follow these rules strictly: ### Layout -- Wrap entire body in a centered `` for consistent margins +- The current shell is a **1200px hybrid-fluid** newsletter (VML hero fallback + stacking media query), not a fixed centered table. Older reports used a centered `
`; the v3 shell supersedes it. Do not hand-roll the shell — reuse the asset's outer structure. - Use **table-based layouts only** — no flexbox, no grid, no float - All styles must be **inline** — Outlook strips ` -Copilot Code Review Effectiveness Analysis + -
+ -

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

+ +
Copilot review effectiveness - engineers adopted 78% of 138 comments; 99.1% precision, only 1 genuine error.
- -

 

Background
+ + -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

+ + + -

2. Review before merging. 38% of ignored comments were on the final commit — the engineer merged without looking at Copilot's last round. Building a habit of checking review comments before clicking merge would give this feedback a chance to have impact.

+ + -

3. Improve the review instructions. The most common dismissal reasons — "lacks domain context," "suggests tests for trivial code," "misunderstands our APIs" — are things we can address by refining our copilot-instructions.md. Adding guidance like "don't suggest tests for telemetry-only changes" or providing context about specific domain patterns could reduce the noise and make the signal-to-noise ratio better for everyone.

+
-

The Android Auth Platform team builds and maintains the authentication libraries used by Microsoft's mobile apps — including MSAL (client-side), Common (shared IPC and utilities), and Broker (the brokered authentication service running on Android devices). Our codebase spans three repositories with 10 active contributors.

+ + -

Earlier this year, we enabled GitHub Copilot code reviews across all three repositories. We also added custom review instructions via copilot-instructions.md to give the AI context about our architecture, coding conventions, and multi-repo structure — aiming to make its feedback more relevant than generic suggestions.

- -

This report answers the question: Is Copilot code review actually useful for our team? We looked at every comment Copilot left over 2 months and determined — through reply analysis, commit diff verification, and AI-assisted classification of ambiguous cases — whether each piece of feedback led to a real code improvement, was dismissed as irrelevant, or was simply never evaluated by the engineer.

+ + - -

 

At a Glance
+ +
+ + + + + + + + + + +
Window 5 weeks Comments 138 Helpful & adopted 78.3% Precision 99.1% View Org-wide (anonymized)
+
- - - - - - - - +
- - -
-
57%
-
of comments received
no response
-
-		 - - -
-
41%
-
confirmed
helpful
-
-		 - - -
-
18%
-
confirmed
not helpful
-
-		 - - -
-
41%
-
unresolved
(inconclusive)
-
-
+
What's New This Cycle
- - - - - - +
Three things that stand out this period
+
 
+
This is our fourth measurement window. Three things stand out from the data.
+
 
+
1. Engineers adopted 78.3% of Copilot's comments outright — a new high. On the code that actually shipped, only 2.2% were left unevaluated. Crucially, 23 adoptions came with no reply and were confirmed against the merged diff (13 by matching Copilot's exact suggestion to the fix commit) — so silent adoption is credited, not lost.
+
 
+
2. We measure what actually merged — plus coverage. The corpus is the 138 comments on human PRs that merged in-window, so every figure describes shipped code. Copilot reviewed 86.4% of merged human PRs (70 of 81). This “what-shipped” basis is a deliberate change from prior cycles, so the rates aren't directly comparable to earlier periods (see Trend).
+
 
+
3. Two numbers — and helpfulness leads. Helpfulness (78.3%) is the headline: how often engineers adopted a comment. Precision (99.1%) backs it up: only one of 138 comments was a genuine error. The gap between them is deliberate declines (18.8%) — correct feedback an engineer consciously chose not to take, not Copilot mistakes.
+
 
+
-The biggest finding isn't about AI quality — it's about adoption. -Of the 557 comments Copilot left, only 239 (43%) received any response from an engineer — the other 318 were never acknowledged. Of those 239 responded-to comments, 144 (60%) led to confirmed code improvements. That's a strong signal: when engineers read and evaluate Copilot's feedback, the majority of it is useful. The challenge is that 57% of comments are never evaluated at all, leaving 226 (41%) in an unresolved state where we can't determine whether they were helpful or not. -
+
At a Glance
- -

Scope

- - - - - - - - +
Helpfulness leads — precision backs it up
+
 
+
We analyzed every inline Copilot comment on the human-authored PRs that merged this period — 138 comments on shipped code.
+
 
+
- - -
-
163
-
Human PRs
scanned
-
-		 - - -
-
113
-
PRs received
Copilot review (69%)
-
-		 - - -
-
557
-
Inline review
comments
-
-		 - - -
-
4.9
-
Avg comments
per reviewed PR
-
-
+ + + + + + + +
+
+
78.3%
+
Confirmed helpful & adopted
+
108 of 138 comments engineers acted on
+
+ 		+
+
99.1%
+
Review precision
+
Only 1 of 109 judged comments was wrong
+
+
+
+
1
+
Genuine error — 0.7%
+
The only slice that reflects on the AI
+
+ 		+
+
81.2%
+
Engineer response rate
+
112 of 138 comments engaged
+
+
- - -

 

Overall Results
- -

Engineer Response Rate

-

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- - - - - - - +
 
+
43% replied57% ignored
+ +
+
How to read this
+
Two numbers, two questions. Helpfulness = did we act on it? (Helpful ÷ all comments = 108 ÷ 138 = 78.3%). Precision = when judged, was Copilot right? (Helpful ÷ (Helpful + Incorrect) = 108 ÷ 109 = 99.1%). A comment can be precise but not helpful — that's the Declined bucket, where Copilot was correct but the engineer made a deliberate call. Of comments engineers actually replied to, ~76% were adopted (85 ÷ 112); the rest were considered declines. That gap is judgment, not error.
+
+
 
+ +
Helpfulness Verdict
- - - - - - - +
The full four-way breakdown
+
 
+
Every comment lands in one of four buckets. Note the red Not helpful segment is barely visible at 0.7% — the only slice that counts against Copilot.
+
 
+
Jun 10 – Jul 15 · 138 comments
 
Engineer replied (239)
 
No response (318)
Helpful 78.3%Declined 18.8%  
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+ + +
+
Formula
+
Precision = Helpful ÷ (Helpful + Incorrect) = 108 ÷ (108 + 1) = 99.1%. Declined and Unresolved are excluded from both sides — neither is a Copilot mistake — so we never penalize the AI for feedback an engineer simply chose not to take.
+
+
 
+ +
By Repository
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- - - - - - - - +
Every repo clears 97% precision
+
 
+
41.3% Helpful18.1%40.6% Unresolved
RepositoryCommentsResponseHelpfulDeclinedIncorrectUnresolvedPrecision
Broker6978.3%52 (75.4%)15 (21.7%)0 (0.0%)2 (2.9%)100%
Common5680.4%45 (80.4%)9 (16.1%)1 (1.8%)1 (1.8%)97.8%
MSAL13100%11 (84.6%)2 (15.4%)0 (0.0%)0 (0.0%)100%
+
 
+
MSAL leads on response (100%) and helpfulness (84.6%) with zero incorrect comments. Broker carries the highest volume and the largest Declined bucket — a team that engages heavily and makes deliberate design calls — and still posts a clean 100% precision. The Incorrect column — the only one that reflects poorly on the AI — is a single comment across all three repos (in Common).
+
 
+ +
By Engineer
- - - - - - - - - +
Declined is not a failure
+
 
+
 
Confirmed Helpful (230)
 
Confirmed Not Helpful (101)
 
Unresolved (226)
EngineerCmtsRespHelpfulDeclinedIncorrectUnresPrecision
Engineer A3070%28200100%
Engineer B2282%14602100%
Engineer C2167%12801100%
Engineer D1573%13200100%
Engineer E14100%12200100%
Engineer F1182%10100100%
Engineer G8100%7100100%
Engineer H5100%401080%
Engineer I5100%3200100%
Engineer J4100%2200100%
Engineer K3100%3000100%
+
 
+ + +
+
Insight
+
The Declined column changes how we read this table. Under the old binary rubric, Engineer C (8 declined) and Engineer B (6 declined) would have looked like the engineers getting the least value from Copilot. In fact both post 100% precision — Copilot was never actually wrong on their PRs. They engage heavily and make frequent, deliberate design-tradeoff calls. Engagement and judgment, not accuracy, are what these numbers really track.
+
+
 
+ +
Trend: Historical Comparison
- - - - - - - - - - - - - - - - - - - - - - - - - - - +
This period vs previous — read across a basis change
+
 
+
One caution before the numbers: this cycle changed how we count. We scoped to merged PRs and credited silent adoption; prior periods counted all comments by creation date. So the quality metrics aren't apples-to-apples — read the pre-June rows as directional only.
+
 
+
Jun 10 – Jul 15 (current basis)
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. All 226 are comments where the engineer did not respond. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
Helpful 78.3%Declined 18.8%  
+
 
+
Apr 20 – Jun 9 (previous basis)
Helpful 52.9%16.6%Unresolved 30.6%
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+
Current vs previous period. Two things you can read cleanly across the basis change: response rate is genuinely up, and precision — the one metric that isolates AI quality — exists from this cycle forward. Rows driven by the basis shift are marked.
+
 
+
MetricPrev · 50dThis · 35dChange
Response rate67.5%81.2%▲ +13.7pp
Replied-helpful rate75.5%75.9%≈ flat
Helpful (of all)52.9%78.3%+25.4pp · basis
Unresolved30.6%2.2%−28.4pp · basis
Copilot precisionn/a99.1%new
+
 
+
The full multi-cycle run — the “Dismissed” column combines Declined + Incorrect for continuity, and the current row (highlighted) is the only one on the merged what-shipped basis:
+
 
+
PeriodDaysCommentsCmts/wkResponseHelpfulDismissedUnresolved
Jan 24 - Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 - Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 - Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 - Jul 153513827.681.2%78.3%19.6%2.2%
+
 
+
Historical “not helpful” rates of 16–21% almost certainly overstated how often the AI was actually wrong — they bundled deliberate declines with genuine errors. Treat pre-June quality as an upper bound, not a fact. The step change at June is largely a definition change, not a behavior change.
+
 
+ +
What Copilot Is Good At
- - - - - - +
Real catches from this period
+
 
+
-226 comments (41%) are unresolved — not because the AI was wrong, but because no one engaged. -All 226 are comments that received no reply from the engineer. This includes 122 comments on the final commit before merge (where the engineer had no subsequent commits), 45 where the file was modified but at different lines, 50 where the comment had no line number for verification, and 9 where the file was never modified. If engineers had responded — even just to dismiss — we would know whether the feedback was useful. -
+
+
Copilot flagged that a flight/telemetry provider was being read before it was initialized — an ordering bug that would silently drop data. “Good catch — this is a fundamental ordering problem. Fixing it.”
+
— Broker PR #215 · AriaInitializer
+
- -

How Helpful Comments Were Delivered

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
+
+
A newly-added overload shadowed a deprecated one; Copilot recommended @JvmOverloads to preserve binary compatibility for Java callers. “Fixed. Thanks for pointing this out.”
+
— Common PR #3147
+
- -

How Not Helpful Comments Were Identified

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
+
+
On an OWASP dependency check, Copilot spotted an inconsistency in how a CVE suppression was applied across modules. “Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951.”
+
— Common PR #3156
+
- - -

 

Results by Repository
- - -

Broker (293 comments)

- - - - - - +
 
+
49%16%35%
+
When Copilot Was Actually Wrong
- - -

Common (188 comments)

- - - - - - +
The one genuine error, in full — all 0.7%
+
 
+
36%18%46%
+
+
Copilot warned a Mockito varargs matcher would misbehave. “Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12.” A version-specific false positive — the engineer's repro proved Copilot wrong. This is the only genuine error in all 138 comments.
+
— Common PR #3171
+
- - -

MSAL (76 comments)

- - - - - - +
 
+
26%25%49%
+ +
+
Measurement integrity
+
The force-push confound. A prior draft nearly counted a second error: a correct Broker @Override comment (PR #212) almost recorded as a hallucination because the engineer force-pushed a rewritten commit ~2 min after opening the PR. Copilot had reviewed the original commit, where the annotation was genuinely absent; the force-push orphaned it, so GitHub marked the note “outdated.” We found this fingerprint on 31 of 138 comments (9 PRs), but only #212 was a proven wrongful-outdating — so 99.1% precision holds. Every “already there / outdated” dismissal on a force-pushed PR is now validated against the immutable reviewed snapshot before it's scored.
+
+
 
+ +
When Copilot Was Right but Declined
- - - - - - - - - - +
Correct feedback, deliberate call — neutral
+
 
+
We hand-audited all 26 Declined comments. Two things stand out: none were Copilot errors — every one was a correct or reasonable note — and all 26 were actively replied to. Each is a deliberate, reviewed call where the engineer read the note and chose otherwise.
+
 
+
 
Helpful
 
Not Helpful
 
Unresolved
+
+
Copilot flagged the comment ~1 per device as potentially misleading. “~ = approximate. i think this is ok.” A fair point the author weighed and consciously declined — a judgment call another reviewer might have taken the other way. Not a Copilot error.
+
— Broker PR #201 · the flagship case
+
- - - - - - - - - - - +
 
+
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
+
+
Copilot suggested not logging a full stack trace for an expected exception. “Since this is a new feature, we want that stack trace for all exceptions.” A sound general heuristic that simply did not apply here.
+
— Broker PR #223
+
- - -

 

Results by Engineer
-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- - - - - - - - - - - - - - - - - - - - - - +
 
+
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
+
What We'd Recommend
- - - - - +
Three opportunities
+
 
+
-Engagement drives value — and visibility. -Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
+ + + +
+
1
+ 		+
Keep engaging — it's working
+
Response rate rose to 81.2% (from 67.5%), and engineers adopted 23 comments silently. A quick reply — even a decline with a one-line reason — is what lets us separate “Copilot was wrong” from “I chose otherwise”; silent adoption we now credit automatically.
+
- - -

 

What Copilot Is Good At
- -

Catching real bugs:

-
-PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers.
-Engineer reply: "You're right. Making the change." -
- -

Stale documentation and naming inconsistencies:

-
-PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant after it was renamed. All four were silently fixed in the next commit. -
- -

CI/pipeline configuration issues:

-
-PR #3038 (Common): Copilot warned that using vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. -
- -

The @copilot apply workflow:

-
-In 16 instances (2.9%), engineers validated the feedback and delegated the fix back: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue. -
- - -

 

What Copilot Struggles With
- -

Lacking domain context:

-
-Copilot: "shared_device_id could be used for tracking — consider hashing before emission."
-Engineer: "The shared_device_id is a random UUID and is not PII." -
- -

Suggesting tests for trivial code:

-
-Copilot: "New telemetry attributes lack test coverage..."
-Engineer: "These are just telemetry related changes and adding unit tests will be overdo here." -
- -

Misunderstanding APIs:

-
-Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."
-Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." -
- - -

 

Key Takeaways
- - - - - - - - - - +
 
+
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality, the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
+ + + +
+
2
+ 		+
Validate “outdated” dismissals before scoring them
+
The force-push confound produced one near-miss this cycle: a correct comment (Broker #212) was almost recorded as a hallucination because a rebase orphaned the reviewed commit. We now validate every “already there / outdated” dismissal on a force-pushed PR against the immutable reviewed snapshot. Engineers can help by not rebasing away the reviewed commit mid-review.
+
- - -

 

How We Measured This
- - - - - - - - +
 
+
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used GitHub compare API to check if suggestion tokens appeared as diff additions, or if exact line ranges were modified.
4. AI-assisted classificationAll 133 ambiguous replies were read and classified by GitHub Copilot, based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong.
+ + + +
+
3
+ 		+
Tune copilot-instructions.md against real false-positive patterns
+
Only one genuine error slipped through (a Mockito 5.11.0 version assumption). The higher-leverage fix is the recurring declined-but-correct nitpicks: teach Broker that telemetry attribute values can be intentional literals, stop flagging approximate “~” rationale comments, and have all repos re-verify a missing @Override/test/#TBD against the PR head before flagging.
+
+
 
+ + +
How We Measured This
+
Methodology notes
+
 
+
Corpus. Every inline Copilot review comment on human-authored PRs that merged between June 10 and July 15, 2026 — 138 comments across Common, MSAL, and Broker. Bot-authored PRs and comments on unmerged or abandoned PRs are excluded, so every number describes code that actually shipped.
+
 
+
Coverage. Copilot reviewed 86.4% of merged human PRs (70 of 81). Most uncovered PRs are trivial (No-Changelog) or predate Copilot review being enabled on that repo.
+
 
+
Four-way verdict. Each comment is hand-classified into exactly one bucket: Helpful (adopted), Declined (correct or reasonable, but the engineer made a deliberate call not to take it), Incorrect (Copilot was factually wrong), or Unresolved (never evaluated). Declined and Incorrect are separated on purpose — only Incorrect counts against the AI.
+
 
+
Silent adoption. Adoption isn't read from replies alone. Where an engineer changed the code without replying, we verify against the merged diff — 13 of 23 silent adoptions were confirmed by matching Copilot's exact suggestion to the fix commit. The diff verification itself is long-standing; what changed this cycle is that we now credit these as Helpful rather than leaving them Unresolved.
+
 
+
Precision. Precision = Helpful ÷ (Helpful + Incorrect). Declined and Unresolved are excluded from both sides, so the metric answers only “when Copilot was judged, was it right?” and never penalizes the AI for feedback an engineer simply chose not to take.
+
 
+
Force-push validation. For every “already there / outdated / not relevant” dismissal we check whether the reviewed commit was later rewritten (original_commit_id vs the PR's current commit list). If it was, we re-derive Copilot's claim from the immutable diff_hunk against the merged state before scoring — so a rebase can't silently record a correct comment as an error (see Broker #212).
+
 
+
Trend caveat. Pre-June periods counted all comments by creation date under a binary rubric; this cycle scopes to merged PRs with the four-way verdict. Cross-period quality comparisons are directional only.
+
 
+
Anonymization. The org-wide edition replaces engineer names with stable Engineer A–K labels ordered by comment volume; all counts are otherwise identical to the team edition.
-

Classification rules: "Confirmed Helpful" requires positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" requires positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage are classified as "Unresolved" rather than assumed unhelpful.

- - -

 

What We'd Recommend
- -

Based on this analysis, we see three opportunities:

-

1. Engage with the feedback. The single biggest improvement would be for engineers to respond to Copilot's review comments — even if just to dismiss them. When engineers engage, 60% of comments are helpful. When they don't, 71% of comments become a black hole of unknown value. A quick "not applicable" is more useful than silence — it tells us whether the AI is giving good feedback, and it builds the data we need to improve the review instructions.

+
+ Generated by the copilot-review-analyst skill · rendered with outlook-highfidelity-report · Merged what-shipped basis (PRs merged Jun 10 – Jul 15, 2026). Methodology: four-way verdict (Helpful / Declined / Incorrect / Unresolved) + Copilot precision, with silent-adoption crediting and force-push-snapshot validation. +
+ - - -
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

- -
+ - - - - - - - - - - - - - - - - - diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html index 72759b00..0fa1f846 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html @@ -98,314 +98,746 @@ .card-grid { grid-template-columns: repeat(2, 1fr); } .donut-container { flex-direction: column; } } + /* --- 4-way methodology additions --- */ + :root { + --blue: #0969da; --light-blue: #ddf4ff; + --purple: #8250df; --light-purple: #fbefff; + } + .card.blue { border-left: 4px solid var(--blue); } + .card.purple { border-left: 4px solid var(--purple); } + .bar-fill.blue { background: var(--blue); } + .bar-fill.purple { background: var(--purple); } + .stacked-segment.green { background: var(--green); } + .stacked-segment.blue { background: var(--blue); } + .stacked-segment.red { background: var(--red); } + .stacked-segment.amber { background: var(--amber); } + .card.six-grid { } + .card-grid.six { grid-template-columns: repeat(6, 1fr); } + @media (max-width: 640px) { .card-grid.six { grid-template-columns: repeat(2, 1fr); } }

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

- - -

Executive Summary

- -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

- -
-
-
57%
-
of comments received
no response
-
-
-
41%
-
confirmed
helpful
-
-
-
18%
-
confirmed
not helpful
-
-
-
41%
-
unresolved
(not evaluated)
-
-
- -
- The biggest finding isn't about AI quality — it's about adoption. - When engineers do engage with Copilot's comments, 60% turn out to be helpful. But 57% of comments receive no response at all, and 41% of all comments are unresolved — we can't tell if they were useful because no one evaluated them. The true helpfulness rate lies between 41% (confirmed floor) and 82% (if all unresolved were helpful). -
- - -

Scope

- -
-
-
163
-
Human PRs
scanned
-
-
-
113
-
PRs received
Copilot review (69%)
-
-
-
557
-
Inline review
comments
-
-
-
4.9
-
Avg comments
per reviewed PR
-
+

Android Auth Platform  |  June 10 – July 15, 2026  |  Common, MSAL, Broker Repositories

+ +

Executive Summary

+

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the 132 comments, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because Copilot was wrong or because the engineer made a deliberate judgment call.

+
+
72%
engineer
response rate
+
54.5%
confirmed
helpful
+
20.5%
correct but
declined
+
1.5%
actually
incorrect
+
23.5%
unresolved
(no evidence)
+
97.3%
review
precision
- - -

Overall Results

- -

Engineer Response Rate

+
Fairness principleWe separate “Declined” (Copilot was correct or reasonable, but the engineer intentionally chose not to act — a nitpick, a stylistic preference, or an approximation the author knew about) from “Incorrect” (Copilot was factually wrong). Only genuinely Incorrect comments count against review quality. Precision = Helpful ÷ (Helpful + Incorrect), excluding Declined and Unresolved — so we never penalize Copilot for feedback it had no way to know the author would decline.
+

This run introduces a new, fairer methodology. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral Declined category and reserves Not Helpful exclusively for comments where Copilot was demonstrably wrong.

+

Key findings:

+
    +
  • 72% of Copilot's comments now receive a response from engineers. This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value.
    • +
  • Only 1.5% of all comments were genuinely wrong. Across 132 comments, just two were confirmed as Copilot errors. When you compute Copilot's precision — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at 97.3%. The AI is rarely incorrect.
    • +
  • 20.5% of comments were correct-but-declined. These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. These do not count against Copilot's quality. Treating them as failures, as the old methodology did, materially understated the tool's accuracy.
    • +
  • 54.5% of all comments led to a confirmed code improvement. With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy.
    • +
+
+

How We Measured This

+

This analysis went through five phases to ensure accuracy:

+
    +

  1. Data collection. We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text.

    +
    2. +

  2. Diff-level verification. For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit.

    +
    3. +

  3. AI-assisted reply classification (three-way). For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts:

    +
      +
    • Helpful — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot).
      • +
    • Declined — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). Copilot was not wrong — it simply had no way to know the author's intent.
      • +
    • Not Helpful (incorrect) — the engineer's reply demonstrates the comment was factually or technically wrong (a false positive, a hallucinated problem, or a misunderstanding of a library/API).
      • +
    +
    4. +

  4. The fairness principle. The dividing line between Declined and Not Helpful is whether Copilot was wrong, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is Declined, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as Not Helpful.

    +
    5. +

  5. Cross-validation. All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved.

    +
    6. +
+

The final dataset classifies each of the 132 comments as confirmed helpful, declined (correct but not adopted), not helpful (incorrect), or unresolved (insufficient evidence to determine). No comment is left without a classification.

+
+

A Note on Fairness: Why "Declined" Is Not "Wrong"

+

The single most important change in this report is the recognition that an engineer declining a comment is not evidence that the comment was bad.

+

Consider Broker PR #201. Copilot flagged a code comment that read ~1 per device as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: "~ = approximate. i think this is ok." Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair.

+

We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise.

+

By separating these Declined cases (20.5% of all comments) from genuine errors (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually wrong in only 2 of 132 comments.

+
+

Overall Results

+ + + + + + + + + + + + + + + + + + + + + + + + + +
MetricValue
Human PRs scanned87
PRs that received Copilot review53 (61%)
Total inline review comments132
Average comments per reviewed PR2.5
+

Engineer Response Rate

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- -
-
43% replied
-
57% ignored
-
-
- Engineer replied (239) - No response (318) -
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- + + + + + + + + + + + + + + + + + + + + +
BehaviorCountPercentage
Engineer replied (any response — acceptance, decline, or discussion)9572.0%
Engineer did not reply3728.0%
+

At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a 100% response rate across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted.

+

Helpfulness Verdict

+

Each comment was classified into one of four categories. Note the deliberate split between Declined (Copilot was right, the engineer chose otherwise) and Not Helpful (Copilot was wrong):

-
41.3%
-
18.1%
-
40.6%
+
54.5%
+
20.5%
+
+
23.5%
- Confirmed Helpful (230) - Confirmed Not Helpful (101) - Unresolved (226) + Helpful (72) + Declined (27) + Incorrect (2) + Unresolved (31)
- - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
VerdictCountPercentageDefinition
Confirmed Helpful7254.5%The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Declined (correct, not adopted)2720.5%The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. This is neutral; Copilot was not wrong.
Not Helpful (incorrect)21.5%The engineer demonstrated the comment was factually or technically wrong — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality.
Unresolved3123.5%The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated.
- -
- 226 comments (41%) are unresolved — not because the AI was wrong, but because no one looked. - This includes 122 comments on the final commit before merge where the engineer had no subsequent commits, 45 where the file was modified at different lines, 50 with no line number for verification, and 9 where the file was never modified. If engineers had engaged — even just to dismiss — we would know. -
- -

How Helpful Comments Were Delivered

+

The Precision Metric

+

Because Declined is neutral and Unresolved is unknown, the fairest single measure of Copilot's review quality is its precision — of the comments we can actually adjudicate as right or wrong, what fraction were right?

+
+

Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%

+
+

Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. 97.3% precision means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy.

+

How Each Category Breaks Down

+

Confirmed Helpful (72):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
PathCountDescription
Engineer replied and acknowledged66Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …")
Engineer silently applied the fix6No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit
- -

How Not Helpful Comments Were Identified

+

Declined — correct but not adopted (27): All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong.

+

Not Helpful — incorrect (2):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
CommentRepo / PRWhy it was wrong
Mockito varargs matcher warningCommon #3171Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12.
"Missing @Override annotation"Broker #212The @Override annotation was already present — a hallucinated problem.
- - -

Results by Repository

- -
-
-
Broker
-
-
48.5%
-
16%
-
35%
-
-
293
-
-
-
Common
-
-
36.2%
-
18%
-
46%
-
-
188
-
-
-
MSAL
-
-
26%
-
25%
-
49%
-
-
76
-
-
-
- Helpful - Not Helpful - Unresolved - (number = total comments) -
- +

Unresolved (31): No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated.

+
+

Results by Repository

- - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
RepositoryCommentsResponse RateHelpfulDeclinedNot HelpfulUnresolvedPrecision
Broker6669.7%33 (50.0%)16 (24.2%)1 (1.5%)16 (24.2%)97.1%
Common5168.6%29 (56.9%)7 (13.7%)1 (2.0%)14 (27.5%)96.7%
MSAL1593.3%10 (66.7%)4 (26.7%)0 (0.0%)1 (6.7%)100%
- -

Broker has the highest response rate (56%) and the highest confirmed helpfulness (49%). In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved.

- - -

Results by Engineer

-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- +

Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository.

+

Coverage across the three repos:

- - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
RepositoryHuman PRsPRs Reviewed by CopilotCoverage
Common382361%
MSAL14750%
Broker352366%
- -
- Engagement drives value — and visibility. - Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
- - -

Response Behavior Deep Dive

- -

What happens when engineers reply? (239 comments)

-
-
60% helpful
-
40% not helpful
-
-

When engineers engage, the majority of Copilot feedback turns out useful. This is the strongest signal that review quality is decent — the bottleneck is adoption.

- -

What happens when engineers don't reply? (318 comments)

-
-
27% silently fixed
-
-
71% unresolved
-
- +
+

Results by Engineer

+

Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break.

- - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code silently applied (verified via diff)5015.7%Confirmed Helpful
Exact commented lines modified in subsequent commit72.2%Confirmed Helpful
Evidence of fix at nearby lines (re-audit)299.1%Confirmed Helpful
Merged without any subsequent commits12238.4%Unresolved
File modified but not at the commented lines4514.2%Unresolved
File modified but no line number to verify5015.7%Unresolved
File never modified after the comment92.8%Unresolved
Comment on stale/outdated code61.9%Confirmed Not Helpful
EngineerCommentsRepliedResponse RateHelpfulDeclinedNot HelpfulUnresolvedHelpfulnessPrecision
Engineer A77100%7000100%100%
Engineer B33100%3000100%100%
Engineer C11981.8%900281.8%100%
Engineer D6583.3%401166.7%80.0%
Engineer E302376.7%1770656.7%100%
Engineer F9555.6%520255.6%100%
Engineer G11654.5%601454.5%85.7%
Engineer H6350.0%300350.0%100%
Engineer I44100%220050.0%100%
Engineer J191578.9%890242.1%100%
Engineer K191578.9%870442.1%100%
Engineer L300%00030%n/a
Engineer M300%00030%n/a
Engineer N100%00010%n/a
- -

The single largest category is the 122 comments (38% of ignored) where the engineer merged the PR without pushing any additional commits after Copilot's review. These comments may have been valid, but we cannot tell because the engineer never evaluated them.

- - -

What Copilot Is Good At

- -

Catching real bugs:

+

Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.

+

Key observation — the Declined column changes how we read this table. Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the least value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a 100% precision: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket.

+
+

Response Behavior Deep Dive

+

Of the 132 total comments:

+
    +
  • 95 (72.0%) received a reply. Of those, 69.5% were helpful, 28.4% were declined (correct but not adopted), and only 2.1% were incorrect. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong.
    • +
  • 37 (28.0%) were not replied to. Of those, 16.2% were silently addressed (verified via diff) and the remaining 83.8% are unresolved — we cannot determine whether the comment was useful because the engineer never evaluated it.
    • +
+

What happens to comments engineers reply to

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Reply verdictCount% of repliedCounts against Copilot?
Helpful — accepted or fixed6669.5%No — positive
Declined — correct but not adopted2728.4%No — neutral
Not Helpful — incorrect22.1%Yes
+

The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot wrong only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently.

+

What happens to ignored comments

+ + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code / exact lines silently applied (verified via diff)616.2%Confirmed Helpful
No reply and no diff evidence either way3183.8%Unresolved
+

The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further.

+
+

What Copilot Is Good At

+

The most valuable helpful comments this period, with real examples from our PRs:

+

Catching real initialization-ordering bugs:

-

PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters.

-

Engineer reply: "You're right. Making the change."

+

PR #215 (Broker) — AriaInitializer: Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."

+

This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone.

- -

Stale documentation and naming inconsistencies:

+

Telemetry schema and contract discrepancies:

-

PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant USE_TEE_ONLY_FOR_TOKEN_BINDING after it was renamed. All four were silently fixed in the next commit.

+

PR #204 (Broker) — AccountChooser: Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +Engineer acknowledged and reconciled the schema.

- -

Dead code and unused imports:

+

Deprecated-API and compatibility issues:

-

PR #3040 (Common): Copilot spotted an unused local variable enabledSettingRaw. Verified as fixed via diff — the suggested replacement code appeared in the commit additions.

+

PR #3147 (Common): Copilot pointed out a newly-added overload shadowed a deprecated one and recommended @JvmOverloads / a default argument to preserve binary compatibility for Java callers. +Engineer reply: "Fixed. Thanks for pointing this out."

- -

CI/pipeline configuration issues:

+

Security and policy consistency:

-

PR #3038 (Common): Copilot warned that vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version.

+

PR #3156 (Common) — OWASP dependency check: Copilot identified an inconsistency in how a CVE suppression was applied across modules. +Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."

+

Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item.

- -

The @copilot apply workflow:

+

Behavior-vs-description correctness:

-

In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back with: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue.

+

PR #3165 (Common) — Authority validation: Copilot noted a URL comparison was matching on the full URL when only the host should matter. +Engineer reply: "Good point — updated the code to check host only."

- - -

What Copilot Struggles With

- -

Lacking domain context:

+
+

When Copilot Was Actually Wrong

+

Only two comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule:

+

Misjudging a library's behavior:

-

Copilot: "shared_device_id could be used for tracking across apps — consider hashing before emission."

-

Engineer: "The shared_device_id is a random UUID generated by one of the participating apps and is not PII."

+

PR #3171 (Common): Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."

+

Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive.

- -

Suggesting tests for trivial code:

+

Hallucinating a missing annotation:

-

Copilot: "New telemetry attributes lack test coverage..."

-

Engineer: "These are just telemetry related changes and adding unit tests will be overdo here."

+

PR #212 (Broker): Copilot claimed a method was missing an @Override annotation. +Engineer reply: "The override annotation is already there."

+

The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero.

- -

Misunderstanding APIs:

+
+

When Copilot Was Right but the Engineer Declined

+

These comments were not wrong — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction.

+

Deliberate notation choice (the flagship case):

-

Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."

-

Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online."

+

PR #201 (Broker): Copilot flagged the code comment ~1 per device as potentially misleading, since the tilde could be read as an exact value. +Engineer reply: "~ = approximate. i think this is ok."

+

A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure.

- -

Commenting on intentional design choices:

+

Subjective naming preference:

-

Copilot: "getPackageInfo() != null is redundant since it either returns PackageInfo or throws..."

-

Engineer: "This is fine. The verbosity makes the code clearer to understand."

+

PR #3151 (Common): Copilot suggested encoding the "prompt" semantics into a span name for clarity. +Engineer reply: "I think the current name is fine."

+

A judgment call on naming. Both names are defensible; the author preferred the existing one.

- - -

Key Takeaways

- - - - - - - - - - - -
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality — explicit engineer dismissals — the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49% helpful), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
8At ~5 comments per PR, the signal-to-noise question can't be settled without engagement. We confirmed ~2 useful comments per PR on average, but with 41% unresolved, the actual number could be higher.
- - -

How We Measured This

- +

Intentional verbosity for a new feature:

+
+

PR #223 (Broker): Copilot suggested not logging a full stack trace for an expected exception type. +Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."

+

The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here.

+
+

Context Copilot couldn't have known:

+
+

PR #3177 (Common): Copilot suggested removing an INFO-level log as noise. +Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."

+

The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff.

+
+
+

Trend Over Time

+

This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the Declined vs. Incorrect split and the Precision metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table.

- - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used the GitHub compare API to check if suggestion tokens appeared as diff additions, or if the exact line range was modified in subsequent commits.
4. AI-assisted classificationAll 133 ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong (e.g., typo fix + matching -1 line file change).
PeriodDaysCommentsComments/wkResponse RateHelpfulDismissed (combined)Unresolved
Jan 24 – Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 – Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 – Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 – Jul 153513226.472.0%51.5%22.0%26.5%
+

What changed this period:

+
    +
  • Response rate rose from 67.5% to 72.0%, driven partly by engineers going back to reply to previously-unanswered comments.
    • +
  • Trend-series helpfulness dipped slightly (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable.
    • +
  • The "Dismissed" number is misleading in isolation. This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that 20.5 of those 22.0 points are Declined (Copilot was right) and only 1.5 points are genuine errors. Copilot's precision this period is 97.3%. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong.
    • +
+

Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.

+

Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.

+
+

Key Takeaways

+
    +

  1. Copilot is almost never wrong — precision is 97.3%. Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74.

    +
    2. +

  2. "Declined" is not "wrong," and it's now counted fairly. 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent.

    +
    3. +

  3. Response rate climbed to 72%. Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments.

    +
    4. +

  4. 54.5% of comments led to a confirmed improvement. With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated.

    +
    5. +

  5. Every repository posts 96%+ precision. MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo.

    +
    6. +

  6. The Declined bucket concentrates in high-engagement engineers. Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem.

    +
    7. +

  7. Unresolved is the only real blind spot. 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report.

    +
    8. +

  8. The bottleneck is adoption and engagement, not accuracy. The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback.

    +
    9. +
+
+

What We'd Recommend

+

Based on this analysis, we see three opportunities:

+

1. Keep engaging — it's working. Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement.

+

2. Close the unresolved gap before merge. The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact.

+

3. Target the two real error patterns. The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining copilot-instructions.md — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero.

+
+

Methodology Notes

+
    +
  • Scope. Only inline code review comments from the Copilot user were counted. PR-level summary comments were excluded from the helpfulness analysis.
    • +
  • Bot exclusions. PRs authored by copilot-swe-agent (Copilot coding agent), dependabot[bot], and github-actions[bot] were excluded. Only PRs authored by human engineers were analyzed.
    • +
  • Three-way reply classification. Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is whether Copilot was factually/technically wrong, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers.
    • +
  • Precision. Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake.
    • +
  • Diff verification. For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as + lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed.
    • +
  • Conservative "Unresolved." Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful."
    • +
  • Account merging. Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings.
    • +
  • Data availability. Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at %TEMP%\copilot-review-analysis\ for independent verification.
    • +
+
+

Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.

-

Classification rules: A comment is "Confirmed Helpful" only with positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" only with positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage — including the final review round merged without evaluation — are classified as "Unresolved" rather than assumed unhelpful.

- -
-
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

-
- + \ No newline at end of file diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md index e0764b36..d4c6dda0 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md @@ -1,19 +1,21 @@ # Copilot Code Review Effectiveness Analysis -**Android Auth Platform | January 23 – March 23, 2026** +**Android Auth Platform | June 10 – July 15, 2026** --- ## Executive Summary -We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past two months. For each of the **557 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs. +We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the **132 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because *Copilot was wrong* or because *the engineer made a deliberate judgment call*. + +This run introduces a **new, fairer methodology**. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral **Declined** category and reserves **Not Helpful** exclusively for comments where Copilot was demonstrably *wrong*. **Key findings:** -- **57% of Copilot's comments received no response from engineers.** This is the single most important number in this report. The majority of AI review feedback is never even acknowledged. -- **Of comments that engineers engaged with, 60% were helpful** — a strong signal that the review quality itself is decent. -- **41% of all comments led to a confirmed code improvement.** But 41% are unresolved — comments that were ignored and where we lack evidence to judge either way. The true helpfulness rate could be significantly higher, but we can't confirm it because no one looked. -- **Engineers who reply to 75%+ of comments see 47-70% helpfulness.** Engineers who reply to <35% see 20-40%. The biggest lever for improving Copilot review value is not the AI — it's engineer engagement with the feedback. +- **72% of Copilot's comments now receive a response from engineers.** This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value. +- **Only 1.5% of all comments were genuinely wrong.** Across 132 comments, just **two** were confirmed as Copilot errors. When you compute Copilot's **precision** — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at **97.3%**. The AI is rarely incorrect. +- **20.5% of comments were correct-but-declined.** These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. **These do not count against Copilot's quality.** Treating them as failures, as the old methodology did, materially understated the tool's accuracy. +- **54.5% of all comments led to a confirmed code improvement.** With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy. --- @@ -21,17 +23,32 @@ We analyzed **every inline code review comment** left by GitHub Copilot on human This analysis went through five phases to ensure accuracy: -1. **Data collection.** We used the GitHub API to extract all 557 Copilot inline review comments from 163 human-authored PRs (excluding PRs authored by Copilot coding agent). We also recorded which comments received human replies and what those replies said. +1. **Data collection.** We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text. -2. **Reply-based classification.** For the 239 comments (43%) that received a human reply, we classified the reply as positive (e.g., "good catch", "fixed", "addressed"), negative (e.g., "won't fix", "not applicable", "by design"), or ambiguous. +2. **Diff-level verification.** For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. -3. **Diff-level verification.** For the 318 comments (57%) that received no reply, we checked whether the engineer acted on the feedback silently. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. +3. **AI-assisted reply classification (three-way).** For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts: + - **Helpful** — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot). + - **Declined** — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). **Copilot was not wrong** — it simply had no way to know the author's intent. + - **Not Helpful (incorrect)** — the engineer's reply demonstrates the comment was factually or technically *wrong* (a false positive, a hallucinated problem, or a misunderstanding of a library/API). -4. **AI-assisted reply classification.** All 133 comments with ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. For example, "this is just telemetry" was classified as dismissed, "@copilot apply changes" was classified as helpful (engineer delegated the fix back to Copilot), and "Added unit test for this" was classified as helpful (engineer acted on the suggestion). These classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. +4. **The fairness principle.** The dividing line between *Declined* and *Not Helpful* is **whether Copilot was wrong**, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is **Declined**, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as **Not Helpful**. -5. **Cross-validation.** All comments initially classified as "not helpful" were re-examined against the diff evidence. 18 were reclassified as helpful where the evidence was strong (e.g., a typo fix suggestion with a corresponding -1 line file change, or an unused import suggestion with a matching -2 line change). +5. **Cross-validation.** All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved. -The final dataset classifies each of the 557 comments as **confirmed helpful**, **confirmed not helpful**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. +The final dataset classifies each of the 132 comments as **confirmed helpful**, **declined (correct but not adopted)**, **not helpful (incorrect)**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. + +--- + +## A Note on Fairness: Why "Declined" Is Not "Wrong" + +The single most important change in this report is the recognition that **an engineer declining a comment is not evidence that the comment was bad.** + +Consider Broker PR #201. Copilot flagged a code comment that read `~1 per device` as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: *"~ = approximate. i think this is ok."* Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair. + +We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise. + +By separating these **Declined** cases (20.5% of all comments) from genuine **errors** (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually *wrong* in only 2 of 132 comments. --- @@ -39,11 +56,10 @@ The final dataset classifies each of the 557 comments as **confirmed helpful**, | Metric | Value | |--------|-------| -| Human PRs scanned | 163 | -| PRs that received Copilot review | 113 (69%) | -| Total inline review comments | 557 | -| PR-level summary comments | 205 | -| Average comments per reviewed PR | 4.9 | +| Human PRs scanned | 87 | +| PRs that received Copilot review | 53 (61%) | +| Total inline review comments | 132 | +| Average comments per reviewed PR | 2.5 | ### Engineer Response Rate @@ -51,220 +67,272 @@ Before looking at helpfulness, it's important to understand how engineers intera | Behavior | Count | Percentage | |----------|-------|------------| -| **Engineer replied** (any response — acceptance, dismissal, or discussion) | 239 | **42.9%** | -| **Engineer did not reply** | 318 | **57.1%** | +| **Engineer replied** (any response — acceptance, decline, or discussion) | 95 | **72.0%** | +| **Engineer did not reply** | 37 | **28.0%** | -More than half of all Copilot review comments receive no human response at all. This means the majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it. +At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a **100% response rate** across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted. ### Helpfulness Verdict -Each comment was classified into one of three categories: +Each comment was classified into one of four categories. Note the deliberate split between **Declined** (Copilot was right, the engineer chose otherwise) and **Not Helpful** (Copilot was wrong): | Verdict | Count | Percentage | Definition | |---------|-------|------------|------------| -| **Confirmed Helpful** | **230** | **41.3%** | The comment led to a code change — either the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | -| **Confirmed Not Helpful** | **101** | **18.1%** | The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design. We have positive evidence that the comment was *wrong*, not merely that it was *ignored*. | -| **Unresolved** | **226** | **40.6%** | The comment received no reply AND we could not confirm whether it was addressed. This includes cases where the engineer merged without any subsequent commits (the final review round was never evaluated), where the file was modified but not at the specific lines Copilot flagged, or where the comment had no line number making verification impossible. | +| **Confirmed Helpful** | **72** | **54.5%** | The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | +| **Declined (correct, not adopted)** | **27** | **20.5%** | The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. **This is neutral; Copilot was not wrong.** | +| **Not Helpful (incorrect)** | **2** | **1.5%** | The engineer demonstrated the comment was factually or technically *wrong* — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality. | +| **Unresolved** | **31** | **23.5%** | The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated. | + +### The Precision Metric + +Because *Declined* is neutral and *Unresolved* is unknown, the fairest single measure of Copilot's review quality is its **precision** — of the comments we can actually adjudicate as right or wrong, what fraction were right? + +> **Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%** -The **unresolved** category is the most important number in this report. These 226 comments — **41% of all feedback** — are not confirmed failures of the AI. They are comments where we simply lack evidence either way because no one engaged with them. Many may be perfectly valid feedback that was never read. If engineers had engaged with them (even just to dismiss), we would know. The true helpfulness rate likely falls somewhere between 41% (confirmed floor) and 82% (if all unresolved were helpful). +Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. **97.3% precision** means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy. ### How Each Category Breaks Down -**Confirmed Helpful (230):** +**Confirmed Helpful (72):** | Path | Count | Description | |------|-------|-------------| -| Engineer replied and acknowledged | 144 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "addressed", "added unit test") | -| Engineer silently applied the fix | 86 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | +| Engineer replied and acknowledged | 66 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …") | +| Engineer silently applied the fix | 6 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | -**Confirmed Not Helpful (101):** +**Declined — correct but not adopted (27):** All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong. -| Path | Count | Description | -|------|-------|-------------| -| Engineer replied and dismissed | 95 | Engineer explicitly explained why the comment was wrong, irrelevant, or by design (e.g., "won't fix", "this is fine", "Copilot is incorrect", "false positive") | -| Comment on code already changed | 6 | Comment was on stale/outdated code that had already been modified in a different commit | +**Not Helpful — incorrect (2):** -**Unresolved (226):** +| Comment | Repo / PR | Why it was wrong | +|---------|-----------|------------------| +| Mockito varargs matcher warning | Common #3171 | Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12. | +| "Missing `@Override` annotation" | Broker #212 | The `@Override` annotation was already present — a hallucinated problem. | -| Path | Count | Description | -|------|-------|-------------| -| Merged without any subsequent commits | 122 | No commits after Copilot's review — the engineer merged the PR without acting on the final review round. The comment may have been valid, but we cannot tell because the engineer never evaluated it. | -| File modified but not at the commented lines | 45 | The file was changed after the review, but the diff shows the changes were at different lines than what Copilot flagged. Possibly addressed via a different approach, or possibly coincidental. | -| File modified but no line number to verify | 50 | Copilot's comment had no line number metadata, and the file was modified. We cannot confirm whether the specific concern was addressed. | -| File never modified after the comment | 9 | The file was not touched in any commit after the review, and no reply was left. The comment may have been valid but was ignored. | +**Unresolved (31):** No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated. --- ## Results by Repository -| Repository | Comments | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | -|------------|----------|---------------|------|------|------| -| **Broker** | 293 | **56.0%** | 142 (48.5%) | 48 (16.4%) | 103 (35.2%) | -| **Common** | 188 | **29.8%** | 68 (36.2%) | 34 (18.1%) | 86 (45.7%) | -| **MSAL** | 76 | **25.0%** | 20 (26.3%) | 19 (25.0%) | 37 (48.7%) | +| Repository | Comments | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Precision | +|------------|----------|---------------|---------|----------|-------------|------------|-----------| +| **Broker** | 66 | **69.7%** | 33 (50.0%) | 16 (24.2%) | 1 (1.5%) | 16 (24.2%) | **97.1%** | +| **Common** | 51 | **68.6%** | 29 (56.9%) | 7 (13.7%) | 1 (2.0%) | 14 (27.5%) | **96.7%** | +| **MSAL** | 15 | **93.3%** | 10 (66.7%) | 4 (26.7%) | 0 (0.0%) | 1 (6.7%) | **100%** | -Broker has the highest response rate (56%) and correspondingly the highest confirmed helpfulness (49%). But even in Broker, 35% of comments are unresolved. In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved, meaning we cannot determine whether the AI's feedback was useful because engineers didn't evaluate it. +Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository. Coverage across the three repos: | Repository | Human PRs | PRs Reviewed by Copilot | Coverage | |------------|-----------|------------------------|----------| -| Common | 68 | 47 | 69% | -| MSAL | 31 | 19 | 61% | -| Broker | 64 | 47 | 73% | +| Common | 38 | 23 | 61% | +| MSAL | 14 | 7 | 50% | +| Broker | 35 | 23 | 66% | --- ## Results by Engineer -Each engineer has two GitHub accounts (a personal account for public repos and an EMU account for the private broker repo). These have been merged. Names are anonymized. +Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break. + +| Engineer | Comments | Replied | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Helpfulness | Precision | +|----------|----------|---------|---------------|---------|----------|-------------|------------|-------------|-----------| +| **Engineer A** | 7 | 7 | **100%** | 7 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer B** | 3 | 3 | **100%** | 3 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer C** | 11 | 9 | **81.8%** | 9 | 0 | 0 | 2 | **81.8%** | 100% | +| **Engineer D** | 6 | 5 | **83.3%** | 4 | 0 | 1 | 1 | **66.7%** | 80.0% | +| **Engineer E** | 30 | 23 | **76.7%** | 17 | 7 | 0 | 6 | **56.7%** | 100% | +| **Engineer F** | 9 | 5 | **55.6%** | 5 | 2 | 0 | 2 | **55.6%** | 100% | +| **Engineer G** | 11 | 6 | **54.5%** | 6 | 0 | 1 | 4 | **54.5%** | 85.7% | +| **Engineer H** | 6 | 3 | **50.0%** | 3 | 0 | 0 | 3 | **50.0%** | 100% | +| **Engineer I** | 4 | 4 | **100%** | 2 | 2 | 0 | 0 | **50.0%** | 100% | +| **Engineer J** | 19 | 15 | **78.9%** | 8 | 9 | 0 | 2 | **42.1%** | 100% | +| **Engineer K** | 19 | 15 | **78.9%** | 8 | 7 | 0 | 4 | **42.1%** | 100% | +| **Engineer L** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer M** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer N** | 1 | 0 | **0%** | 0 | 0 | 0 | 1 | **0%** | n/a | + +*Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.* + +**Key observation — the Declined column changes how we read this table.** Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the *least* value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a **100% precision**: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket. -| Engineer | Comments | Replied | Ignored | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | Helpfulness | -|----------|----------|---------|---------|---------------|------|------|------|-------------| -| **Engineer A** | 20 | 20 | 0 | **100%** | 14 | 6 | 0 | **70.0%** | -| **Engineer B** | 83 | 75 | 8 | **90.4%** | 57 | 26 | 0 | **68.7%** | -| **Engineer C** | 15 | 4 | 11 | **26.7%** | 8 | 2 | 5 | **53.3%** | -| **Engineer D** | 40 | 30 | 10 | **75.0%** | 19 | 14 | 7 | **47.5%** | -| **Engineer E** | 110 | 37 | 73 | **33.6%** | 44 | 18 | 48 | **40.0%** | -| **Engineer F** | 99 | 20 | 79 | **20.2%** | 36 | 11 | 52 | **36.4%** | -| **Engineer G** | 63 | 22 | 41 | **34.9%** | 20 | 12 | 31 | **31.7%** | -| **Engineer H** | 100 | 24 | 76 | **24.0%** | 27 | 12 | 61 | **27.0%** | -| **Engineer I** | 24 | 6 | 18 | **25.0%** | 5 | 0 | 19 | **20.8%** | -| **Engineer J** | 3 | 1 | 2 | **33.3%** | 0 | 0 | 3 | **0%** | +--- -*Helpfulness = Confirmed Helpful / Total Comments. Response Rate = Replied / Total Comments. Unresolved = comments with no reply and no definitive diff evidence either way.* +## Response Behavior Deep Dive -**Key observation:** There is a strong correlation between response rate and helpfulness. Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness (70% and 69%) — and crucially, **zero unresolved comments**. When engineers engage, we know exactly what's helpful and what's not. Engineers with low response rates (Engineer F at 20%, Engineer H at 24%) have massive unresolved buckets (52 and 61 comments respectively) — over half their comments go into a black hole where we can't tell if the AI was right or wrong. +Of the 132 total comments: ---- +- **95 (72.0%) received a reply.** Of those, **69.5% were helpful**, **28.4% were declined** (correct but not adopted), and only **2.1% were incorrect**. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong. +- **37 (28.0%) were not replied to.** Of those, **16.2% were silently addressed** (verified via diff) and the remaining **83.8% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. -## Response Behavior Deep Dive +### What happens to comments engineers reply to -Of the 557 total comments: +| Reply verdict | Count | % of replied | Counts against Copilot? | +|---------------|-------|-------------|-------------------------| +| Helpful — accepted or fixed | 66 | 69.5% | No — positive | +| Declined — correct but not adopted | 27 | 28.4% | **No — neutral** | +| Not Helpful — incorrect | 2 | 2.1% | Yes | -- **239 (42.9%) received a reply.** Of those, **60.3% were helpful** and 39.7% were not helpful. When engineers engage, the majority of Copilot feedback turns out to be useful. -- **318 (57.1%) were ignored.** Of those, **27.0% were silently addressed** (verified via diff), **1.9% were on stale code** (confirmed not helpful), and the remaining **71.1% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. +The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot *wrong* only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently. ### What happens to ignored comments | What happened | Count | % of ignored | Verdict | |---------------|-------|-------------|---------| -| Suggestion code silently applied (verified via diff) | 50 | 15.7% | Confirmed Helpful | -| Exact commented lines modified in subsequent commit | 7 | 2.2% | Confirmed Helpful | -| Re-audit: evidence of fix at nearby lines | 29 | 9.1% | Confirmed Helpful | -| Merged without any subsequent commits | 122 | 38.4% | **Unresolved** | -| File never modified after the comment | 9 | 2.8% | **Unresolved** | -| Comment on stale/outdated code | 6 | 1.9% | Confirmed Not Helpful | -| File modified but not at the commented lines | 45 | 14.2% | **Unresolved** | -| File modified but no line number to verify | 50 | 15.7% | **Unresolved** | +| Suggestion code / exact lines silently applied (verified via diff) | 6 | 16.2% | Confirmed Helpful | +| No reply and no diff evidence either way | 31 | 83.8% | **Unresolved** | -The single largest category is the **122 comments (38.4% of ignored)** where the engineer merged the PR without pushing any additional commits after Copilot's review. These represent the final review round being skipped entirely — the feedback had zero chance of impact regardless of its quality. +The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further. --- ## What Copilot Is Good At -The most common categories of helpful comments, with real examples from our PRs: +The most valuable helpful comments this period, with real examples from our PRs: -**Catching real bugs:** -> *PR #3050 (Common):* Copilot flagged that `"$it"` string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters. -> *Engineer reply: "You're right. Making the change."* +**Catching real initialization-ordering bugs:** +> *PR #215 (Broker) — `AriaInitializer`:* Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +> *Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."* +> +> This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone. -**Stale documentation and naming inconsistencies:** -> *PR #64 (Broker):* Copilot identified four locations where KDoc still referenced the old flight constant `USE_TEE_ONLY_FOR_TOKEN_BINDING` after it was renamed to `USE_TEE_ONLY_FOR_HARDWARE_BOUND_KEYS`. All four were silently fixed in the next commit. +**Telemetry schema and contract discrepancies:** +> *PR #204 (Broker) — `AccountChooser`:* Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +> *Engineer acknowledged and reconciled the schema.* -**Dead code and unused imports:** -> *PR #3040 (Common):* Copilot spotted an unused local variable `enabledSettingRaw` that was assigned but never read. Verified as fixed via diff analysis — the suggested replacement code appeared in the commit additions. +**Deprecated-API and compatibility issues:** +> *PR #3147 (Common):* Copilot pointed out a newly-added overload shadowed a deprecated one and recommended `@JvmOverloads` / a default argument to preserve binary compatibility for Java callers. +> *Engineer reply: "Fixed. Thanks for pointing this out."* -**CI/pipeline configuration issues:** -> *PR #3038 (Common):* Copilot warned that using `vmImage: 'windows-latest'` makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. +**Security and policy consistency:** +> *PR #3156 (Common) — OWASP dependency check:* Copilot identified an inconsistency in how a CVE suppression was applied across modules. +> *Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."* +> +> Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item. -**The `@copilot apply` workflow:** -> In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back to Copilot with: `@copilot open a new pull request to apply changes based on [this feedback]`. This is an efficient pattern where AI identifies and fixes the issue end-to-end. +**Behavior-vs-description correctness:** +> *PR #3165 (Common) — Authority validation:* Copilot noted a URL comparison was matching on the full URL when only the host should matter. +> *Engineer reply: "Good point — updated the code to check host only."* --- -## What Copilot Struggles With +## When Copilot Was Actually Wrong -The most common categories of unhelpful comments, with real examples: +Only **two** comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule: -**Lacking domain context:** -> *Copilot:* "`shared_device_id` could be used for tracking across apps — consider hashing before emission." -> *Engineer:* "The shared_device_id is a random UUID generated by one of the participating apps and is not PII." +**Misjudging a library's behavior:** +> *PR #3171 (Common):* Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +> *Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."* > -> Copilot applied a general security heuristic without understanding that this particular identifier is already random and non-linkable. +> Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive. -**Suggesting tests for trivial code:** -> *Copilot:* "New telemetry attributes lack test coverage..." -> *Engineer:* "These are just telemetry related changes and adding unit tests will be overdo here." +**Hallucinating a missing annotation:** +> *PR #212 (Broker):* Copilot claimed a method was missing an `@Override` annotation. +> *Engineer reply: "The override annotation is already there."* > -> This was a recurring theme — Copilot frequently requests tests for logging/telemetry code that the team considers low-risk. +> The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero. -**Misunderstanding APIs:** -> *Copilot:* "`00000003-0000-0ff1-ce00-000000000000` is the resource ID for Microsoft Graph, not SharePoint Online." -> *Engineer:* "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." +--- + +## When Copilot Was Right but the Engineer Declined + +These comments were **not wrong** — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction. + +**Deliberate notation choice (the flagship case):** +> *PR #201 (Broker):* Copilot flagged the code comment `~1 per device` as potentially misleading, since the tilde could be read as an exact value. +> *Engineer reply: "~ = approximate. i think this is ok."* +> +> A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure. + +**Subjective naming preference:** +> *PR #3151 (Common):* Copilot suggested encoding the "prompt" semantics into a span name for clarity. +> *Engineer reply: "I think the current name is fine."* > -> Copilot was factually wrong about a well-known Microsoft service resource ID. +> A judgment call on naming. Both names are defensible; the author preferred the existing one. -**Commenting on intentional design choices:** -> *Copilot:* "`getPackageInfo() != null` is redundant since it either returns PackageInfo or throws..." -> *Engineer:* "This is fine. The verbosity makes the code clearer to understand." +**Intentional verbosity for a new feature:** +> *PR #223 (Broker):* Copilot suggested not logging a full stack trace for an expected exception type. +> *Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."* +> +> The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here. -**Over-engineering suggestions:** -> *Copilot:* "Use a bounded min-heap of size `PRT_ARTIFACT_LIMIT` to reduce overhead..." -> *Engineer:* (The list can only ever contain 3 items — there are only 3 broker apps on Android.) +**Context Copilot couldn't have known:** +> *PR #3177 (Common):* Copilot suggested removing an INFO-level log as noise. +> *Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."* +> +> The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff. --- -## Most Reviewed Files +## Trend Over Time + +This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the **Declined vs. Incorrect** split and the **Precision** metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table. + +| Period | Days | Comments | Comments/wk | Response Rate | Helpful | Dismissed (combined) | Unresolved | +|--------|------|----------|-------------|---------------|---------|----------------------|------------| +| Jan 24 – Mar 25 | 60 | 570 | 66.3 | 44.4% | 38.6% | 17.4% | 44.0% | +| Mar 24 – Apr 19 | 26 | 170 | 45.9 | 78.8% | 62.9% | 21.2% | 15.9% | +| Apr 20 – Jun 9 | 50 | 157 | 22.1 | 67.5% | 52.9% | 16.6% | 30.6% | +| **Jun 10 – Jul 15** | **35** | **132** | **26.4** | **72.0%** | **51.5%** | **22.0%** | **26.5%** | -| Rank | File | Comments | -|------|------|----------| -| 1 | `.github/workflows/copilot-issue-response.yml` | 27 | -| 2 | `broker4j/.../AttributeName.java` | 19 | -| 3 | `.github/workflows/copilot-ci-feedback.md` | 16 | -| 4 | `common/.../AuthorizationFragment.java` | 15 | -| 5 | `broker4j/.../MultipleWorkplaceJoinDataStore.java` | 11 | -| 6 | `broker4j/.../AbstractBrokerController.java` | 11 | -| 7 | `common/.../AzureActiveDirectoryWebViewClient.java` | 11 | -| 8 | `AADAuthenticator/.../BrowserSsoProvider.kt` | 10 | -| 9 | `broker4j/.../BrokerFlight.java` | 9 | -| 10 | `broker4j/.../DeviceRegistrationRequestHandler.java` | 9 | +**What changed this period:** -CI/workflow files and large Java classes with many change touchpoints attract the highest volume of comments. `AttributeName.java` appears frequently because many PRs add telemetry attributes. +- **Response rate rose from 67.5% to 72.0%**, driven partly by engineers going back to reply to previously-unanswered comments. +- **Trend-series helpfulness dipped slightly** (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable. +- **The "Dismissed" number is misleading in isolation.** This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that **20.5 of those 22.0 points are Declined (Copilot was right)** and only **1.5 points are genuine errors**. Copilot's **precision this period is 97.3%**. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong. + +*Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.* + +*Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.* --- ## Key Takeaways -1. **57% of Copilot review comments receive no response from engineers.** This is the most significant finding. The majority of AI review feedback is never acknowledged. Some of it is silently acted on (27% of ignored comments show diff evidence of a fix), but the vast majority — 71% of ignored comments — are unresolved. We simply don't know if they were useful because no one evaluated them. +1. **Copilot is almost never wrong — precision is 97.3%.** Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74. + +2. **"Declined" is not "wrong," and it's now counted fairly.** 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent. + +3. **Response rate climbed to 72%.** Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments. + +4. **54.5% of comments led to a confirmed improvement.** With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated. -2. **41% of all comments led to a confirmed code improvement.** But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement. +5. **Every repository posts 96%+ precision.** MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo. -3. **When engineers engage, 60% of comments are helpful.** Of the 239 comments that received a reply, 144 (60%) led to acknowledged improvements. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy. +6. **The Declined bucket concentrates in high-engagement engineers.** Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem. -4. **Only 18% of comments are confirmed not helpful.** When we restrict "not helpful" to comments where the engineer explicitly dismissed the feedback — the only cases where we have positive evidence of low quality — the rate is surprisingly low. +7. **Unresolved is the only real blind spot.** 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report. -5. **Engagement is the strongest predictor of value.** Engineers who reply to 75%+ of comments see 47-70% confirmed helpfulness with zero unresolved comments. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets (50-60% of their comments). The tool works better when engineers work with it. +8. **The bottleneck is adoption and engagement, not accuracy.** The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback. -6. **38% of ignored comments are on the final commit before merge.** These 122 comments represent the last review round being skipped entirely — the engineer merged without pushing any further changes. These comments may have been perfectly valid, but the feedback had zero chance of impact. +--- + +## What We'd Recommend + +Based on this analysis, we see three opportunities: -7. **Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%).** This correlates with response rate: Broker engineers reply to 56% of comments, while Common (30%) and MSAL (25%) reply far less. +**1. Keep engaging — it's working.** Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement. -8. **At ~5 comments per PR, the signal-to-noise conversation can't be settled without engagement.** We confirmed ~2 useful comments per PR on average, but with 41% of comments unresolved, the actual number could be higher. The only way to know is for engineers to evaluate the feedback. +**2. Close the unresolved gap before merge.** The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact. + +**3. Target the two real error patterns.** The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining `copilot-instructions.md` — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero. --- ## Methodology Notes -- **Scope.** Only inline code review comments from the `Copilot` and `copilot-pull-request-reviewer[bot]` users were counted. PR-level summary comments (205 total) were excluded from the helpfulness analysis. -- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent) were excluded. Only PRs authored by human engineers were analyzed. -- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` (addition) lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped with the comment's target line range (±5 line tolerance). This is a conservative approach — some fixes that refactored code differently than suggested may be missed. -- **AI-assisted reply classification.** Every comment that could not be definitively classified by automated methods was individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. These AI classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. -- **Conservative approach.** We only classify a comment as "Confirmed Helpful" when there is positive evidence: an explicit engineer acknowledgment, or verified code changes at the exact lines/tokens suggested. We only classify as "Confirmed Not Helpful" when there is positive evidence that the comment was *wrong or irrelevant*: an explicit engineer dismissal with a stated reason. Comments where the engineer simply did not engage — including the final review round that was merged without evaluation, files that were never modified, and files modified at different lines — are classified as "Unresolved" rather than assumed to be unhelpful. -- **Account merging.** Engineers with separate public GitHub accounts and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. -- **Data availability.** Raw data for all 557 comments (including comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. +- **Scope.** Only inline code review comments from the `Copilot` user were counted. PR-level summary comments were excluded from the helpfulness analysis. +- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent), `dependabot[bot]`, and `github-actions[bot]` were excluded. Only PRs authored by human engineers were analyzed. +- **Three-way reply classification.** Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is **whether Copilot was factually/technically wrong**, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers. +- **Precision.** Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake. +- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed. +- **Conservative "Unresolved."** Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful." +- **Account merging.** Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. +- **Data availability.** Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. --- -*Analysis conducted March 23-24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.* +*Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.* + diff --git a/.github/skills/copilot-review-analyst/references/classification-rules.md b/.github/skills/copilot-review-analyst/references/classification-rules.md index 7685b431..644af53e 100644 --- a/.github/skills/copilot-review-analyst/references/classification-rules.md +++ b/.github/skills/copilot-review-analyst/references/classification-rules.md @@ -4,10 +4,25 @@ Guide for the AI agent performing Phase 3 reply classification. Read each replie ## Phase 3: Classifying Replied Comments -For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of: +For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of **three** verdicts: - **`helpful`** — The engineer's reply indicates Copilot's feedback led to (or will lead to) a code improvement -- **`not-helpful`** — The engineer's reply indicates Copilot's feedback was wrong, irrelevant, or not actionable +- **`not-helpful`** — Copilot's feedback was **factually wrong** (a false positive, hallucination, or a demonstrable error about the code/library/framework). This is the only bucket that counts *against* Copilot's review quality. +- **`declined`** — Copilot's feedback was **correct or reasonable**, but the engineer **intentionally chose not to act** on it for a by-design, subjective, or contextual reason (or the code became moot). This is **neutral** — it is *not* counted against Copilot, because Copilot had no way to know the author's intent beforehand. + +### Why `declined` exists (the fairness principle) + +A dismissal is **not** evidence that Copilot was wrong. Copilot reviews a diff without the author's design intent, offline discussions, or downstream context. When an engineer replies "intentional" or "this is fine as-is," Copilot's observation was often perfectly valid — the engineer simply made a judgment call. Counting these against Copilot punishes it for not being a mind-reader and understates real review quality. + +So we split the old "not-helpful" bucket into two: +- **`not-helpful` (incorrect)** — Copilot was actually *wrong*. Fair to count against it. +- **`declined` (by-design / subjective / moot)** — Copilot was *right or reasonable*, engineer declined. Neutral. + +The **Copilot precision** metric measures correctness only where it is evaluable: + +> **precision = helpful ÷ (helpful + not-helpful-incorrect)** + +`declined` and `unresolved` are excluded from both numerator and denominator. Precision answers: *"When Copilot spoke and we could judge correctness, how often was it actually right?"* ### What counts as Helpful @@ -18,26 +33,88 @@ For every comment where `HasReply = true`, read the `CommentBody` (what Copilot - **Linked a commit**: Reply contains a commit SHA or link showing they applied a fix - **Acknowledged for future**: "this can be considered in another PR" (acknowledges the issue is valid) -### What counts as Not Helpful +### What counts as Not Helpful (Copilot incorrect) + +Reserve this bucket for cases where the engineer's reply demonstrates Copilot was **factually wrong**: -- **Explicit dismissal**: "won't fix", "by design", "intentional", "not applicable", "false positive", "not relevant" -- **Copilot was wrong**: "incorrect", "Copilot is wrong", "hallucinating", "not accurate", "misunderstanding" -- **Already handled**: "already done", "already handled", "this is fine" -- **Explained away**: Engineer explains why the suggestion doesn't apply — "this is just telemetry", "we consciously chose this", "legacy code", "overdo", "only used in X context", "can't happen" -- **Dismissed or outdated**: "outdated", "dismissed", "out of scope" +- **Disproven claim**: Engineer verified the opposite — "Verified this isn't reproducible on Mockito 5.11.0; the suite passes 12/12" (Copilot was wrong about library behavior) +- **Hallucination**: Copilot flagged something that isn't there — "The `@Override` annotation is already there" (Copilot claimed it was missing) +- **Wrong about the code/framework/version**: Copilot's premise is objectively false for this codebase +- **False positive with proof**: Engineer shows the flagged issue cannot occur + +If the engineer *demonstrates* Copilot got a fact wrong → `not-helpful`. + +### What counts as Declined (neutral — Copilot correct/reasonable, engineer declined) + +- **By-design**: "intentional", "by design", "this is intentional", "we consciously chose this", "keeping as-is" +- **Subjective / nitpick**: "too nitpicky", "the current name is fine", "minor, no-fix", "imo this reads fine" +- **Contextual reason to keep**: "we need this log to catch a regression in cobo/cope", "for a new feature we want the full stack trace" +- **Acceptable risk**: "this should be okay since it's in a log file" +- **Moot / obviated**: "not relevant now that we removed this change", "outdated" (the code changed so the comment no longer applies — Copilot wasn't *wrong*, just stale) +- **Deprecated target**: "this config is deprecated, not fixing" +- **Accurate but declined**: engineer explicitly says the comment *was* accurate but chose not to act ("this is outdated, but was accurate") + +The test: *Did the engineer demonstrate Copilot was wrong (→ not-helpful), or did they acknowledge/ignore a valid-but-unwanted point (→ declined)?* When in doubt between `not-helpful` and `declined`, prefer **`declined`** unless there is positive evidence Copilot was factually incorrect. ### Edge Cases - **Mixed signals** (both positive and negative in same reply): Read the full reply to determine the engineer's overall intent. Don't rely on individual words — understand the sentence. -- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **not-helpful** if they're brushing it off. -- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer whether the engineer is acknowledging or dismissing. When genuinely unclear, lean toward **not-helpful** (conservative). +- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **declined** if they're setting it aside as by-design/out-of-scope (not because Copilot was wrong). +- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer intent. If it reads as acknowledgment → helpful. If it reads as a soft dismissal of a valid point → declined. Only use **not-helpful** when there is evidence Copilot was factually wrong. +- **"Outdated"**: Treat as **declined** (the code moved on; not a Copilot correctness failure) unless the engineer indicates Copilot's underlying claim was itself wrong. **Before scoring, apply the Force-Push Confound check below** — an "outdated" marker on a rewritten snapshot can hide a comment that was *correct* when Copilot reviewed it, and must never be recorded as `not-helpful`. ### Important: Read the Full Reply Do NOT use simple keyword matching. Read the engineer's full reply in context. For example: - "This won't fix the actual issue we're seeing" — This is NOT a "won't fix" dismissal; the engineer is discussing a different topic -- "Thanks but this is intentional" — Despite "thanks", this is a dismissal +- "Thanks but this is intentional" — Despite "thanks", this is a **declined** (by-design), not a Copilot error - "I disagree with this specific suggestion but good catch on the typo above" — Mixed; classify based on the primary concern +- "~ = approximate, I think this is ok" — Copilot's readability nit was reasonable; engineer made a judgment call → **declined**, not not-helpful + +## The Force-Push Confound (mandatory validation before scoring) + +**A "correct" Copilot comment can look like a false positive when the engineer rewrites history.** If an engineer rebases or force-pushes *after* Copilot reviews — a common habit right after opening a PR — the commit Copilot reviewed is orphaned. GitHub then marks the comment **"outdated"** and the visible code no longer matches what Copilot saw, so a comment that was *accurate at the reviewed snapshot* reads as stale or even hallucinated. Left unchecked, this silently inflates the `declined`/`unresolved` buckets and, in the worst case, scores a **correct** comment as `not-helpful` — understating precision. + +> This was not hypothetical: it caused the single decisive misclassification in the Jul 2026 audit (broker #212's `@Override` comment was mislabeled a hallucination; the `@Override` was genuinely absent at the reviewed commit `bcf360ea`, which the engineer force-pushed away ~2 min later). Precision moved **up** once corrected. + +### Detection — three immutable fields + +Phase 1 (`analyze.ps1`) now records these per comment in `raw_results.json`: + +- **`OriginalCommitId`** — `original_commit_id`, the exact commit Copilot reviewed. Immutable; survives force-push. +- **`ReviewedCommitRewritten`** — `true` when `OriginalCommitId` is **absent** from the PR's current commit list (`/pulls/{n}/commits`), i.e. the reviewed snapshot was rewritten. This is the force-push fingerprint. +- **`DiffHunk`** — `diff_hunk`, a frozen copy of the exact lines Copilot saw. Ground truth for what Copilot was actually looking at. + +`precise.ps1` carries `ReviewedCommitRewritten` forward and flags any `not-applied`/`file-not-changed` verdict that sits on a rewritten snapshot. + +### The rule + +For **every** comment classified `declined`, `not-helpful`, `unresolved`, or a no-reply `not-applied`/`file-not-changed` verdict, **first check `ReviewedCommitRewritten`**: + +1. If `false` → classify normally (no confound). +2. If `true` → the dismissal is **unsafe to credit as-is**. Re-derive Copilot's claim from the immutable `DiffHunk` (and, if needed, the reviewed file via `contents?ref=`) and compare it to the **merged** state: + - Copilot's claim was **true at the reviewed snapshot** and the merged code already satisfies it → the fix was effectively adopted by the rewrite → **helpful** (never `not-helpful`, never `declined`). + - Copilot's claim was **true at the reviewed snapshot** and the engineer explicitly declined it on the merits ("intentional", "okay in a log file") → **declined** (the rewrite was incidental). + - Copilot's claim was **false even at the reviewed snapshot** → `not-helpful` (a genuine error). + +**Never assign `not-helpful` to a comment on a rewritten snapshot without confirming Copilot was wrong at `OriginalCommitId`.** A stale marker is not evidence of a Copilot error. + +### Precision is computed only after this validation + +Because a mis-scored force-push case lands directly in the precision denominator, **precision must be recomputed only after the force-push validation pass has run** over all `declined`/`not-helpful`/`unresolved` comments. Report the count of `ReviewedCommitRewritten = true` comments as a measurement-integrity note. + +### Reproducible check (per comment) + +```bash +# 1. The commit Copilot reviewed (immutable): +gh api repos//pulls/comments/ --jq '.original_commit_id' +# 2. Is it still in the PR's commit list? (empty output => rewritten away => confound) +gh api repos//pulls//commits --paginate --jq '.[].sha' | grep +# 3. Ground truth Copilot saw: +gh api repos//pulls/comments/ --jq '.diff_hunk' +# 4. (optional) The reviewed file as Copilot saw it: +gh api "repos//contents/?ref=" --jq '.content' | base64 -d +``` ## Phase 3 Output Format @@ -51,11 +128,11 @@ Map of comment ID to verdict for every replied comment: { "1234567890": "helpful", "1234567891": "not-helpful", - "1234567892": "helpful" + "1234567892": "declined" } ``` -Keys are comment IDs (as strings). Values are `"helpful"` or `"not-helpful"`. +Keys are comment IDs (as strings). Values are `"helpful"`, `"not-helpful"` (Copilot incorrect), or `"declined"` (Copilot correct/reasonable but engineer declined). ### `reaudit-flips.json` @@ -81,7 +158,9 @@ These are handled by `precise.ps1` and `final-classification.ps1` automatically: | `suggestion-applied`, `suggestion-likely-applied`, `exact-lines-modified` | **Helpful** | | `lines-modified-different-fix` | **Helpful** | | `file-changed-elsewhere`, `file-changed-no-line-info` | **Not Helpful** (unless in re-audit flips) | -| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** | +| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** (unless in re-audit flips) | + +> **Force-push caveat:** any `not-applied` / `file-not-changed` row with `ReviewedCommitRewritten = true` is UNSAFE to score as Not Helpful — the reviewed snapshot was rewritten, so the fix may live in the orphaned commit. Validate against the immutable `DiffHunk` (see "The Force-Push Confound") before finalizing; flip to **Helpful** if Copilot's claim was already satisfied by the merged code. ## Account Mapping diff --git a/.github/skills/copilot-review-analyst/references/report-formatting.md b/.github/skills/copilot-review-analyst/references/report-formatting.md index b74db5b4..00961c2f 100644 --- a/.github/skills/copilot-review-analyst/references/report-formatting.md +++ b/.github/skills/copilot-review-analyst/references/report-formatting.md @@ -11,18 +11,20 @@ Generate both formats. Templates are in `assets/` within this skill folder. | # | Section | Content | Depth | |---|---------|---------|-------| | 1 | **Background** | Team context, what repos are covered, what was enabled | 2-3 prose paragraphs | -| 2 | **At a Glance** | 4 summary cards (no-response %, helpful %, not-helpful %, unresolved %) + callout about adoption | Cards + 1 detailed callout box | -| 3 | **Overall Results** | Response rate bar, helpfulness verdict bar, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + 2 breakdown tables | -| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/not/unresolved) | Bar per repo + data table + 1 interpretive paragraph | +| 2 | **At a Glance** | 5 summary cards (helpful %, declined %, incorrect %, unresolved %, precision) + callout explaining the fairness correction | Cards + 1 detailed callout box | +| 3 | **Overall Results** | Response rate bar, four-way verdict bar, precision formula, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + breakdown tables | +| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/declined/incorrect/unresolved/precision) | Bar per repo + data table + 1 interpretive paragraph | | 5 | **Results by Engineer** | Table with colored columns (anonymize names for org-wide sharing) | Full table + blue callout box highlighting the engagement-value correlation | | 6 | **Response Behavior Deep Dive** | What happens to ignored comments (silently applied, merged without commits, etc.) | Summary stats + detailed breakdown table + interpretive paragraph | | 7 | **What Copilot Is Good At** | 4-5 real examples with PR references and engineer quotes | Each example: category header + full Copilot comment text (not truncated) + engineer reply + 1-2 sentence explanation | -| 8 | **What Copilot Struggles With** | 4-5 real examples showing false positives, domain gaps | Same format as above — full quotes + explanatory context | +| 8 | **When Copilot Was Wrong / When Feedback Was Declined** | Genuine errors and correct-but-declined examples in separate subsections | Full quotes + explanatory context; never present a deliberate decline as a Copilot error | | 9 | **Most Reviewed Files** | Top 10 files by comment count | Table + 1 interpretive paragraph | | 10 | **Key Takeaways** | 7-8 numbered findings | Each finding: bold stat + explanatory sentence | | 11 | **Recommendations** | 3 actionable next steps | Each recommendation: 1 full prose paragraph (not a bullet point) with reasoning | | 12 | **Methodology Notes** | How data was collected, classified, and validated | 5-6 bullet points with sufficient detail for reproducibility | +**Headline framing (all formats).** The hero band and the first Key Takeaway lead with **Helpful & adopted %** (adoption), *not* precision. Adoption answers "did engineers act on the feedback?" — the question leadership cares about — and precision is presented as the supporting second number. Leading with precision (which is near-100% by construction once Declined/Unresolved are excluded) overstates the story and buries the adoption signal. + ## Trend Section (Section 2.5 — between At a Glance and Overall Results) **Only generated when `history.json` has ≥2 entries.** Skip entirely on the first run. @@ -35,11 +37,12 @@ Load `~/.copilot-review-analysis/history.json`. Entries are sorted newest-first. Since periods may have different lengths (e.g., 60 days vs 14 days): -1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, not-helpful %, unresolved %, and replied-helpful rate are directly comparable across any period length. +1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, combined dismissal %, unresolved %, and replied-helpful rate are directly comparable across any period length. For snapshots created after the three-way methodology change, also show Declined %, Incorrect %, and Precision. 2. **Show counts as context only.** Display total comments alongside `comments/week` for normalized volume comparison. Do NOT compute count deltas like "comments dropped from 570 to 85" — this is misleading when periods differ. 3. **Show period duration prominently.** Every trend row must include the date range and duration (e.g., "Jan 24–Mar 25 (60d)"). 4. **Use "pp" (percentage points) for deltas.** "↑ +7.6pp" not "↑ +7.6%". The delta is the arithmetic difference between two percentages. 5. **Color-code deltas.** Green (↑) for improvements (response rate up, helpful up, not-helpful down, unresolved down). Red (↓) for regressions. +6. **Preserve trend continuity.** Historical snapshots created before the `declined` verdict only have a combined dismissal rate. Label that column **Dismissed (combined)** and explain that the current period can be split into Declined (neutral) and Incorrect (counts against Copilot). Do not retroactively infer the split for old periods. ### Markdown Format (2 runs — current vs previous) @@ -98,10 +101,19 @@ From `final_classification.json`: # Overall $total = $data.Count $helpful = ($data | Where-Object { $_.Verdict -eq "helpful" }).Count -$notHelpful = ($data | Where-Object { $_.Verdict -eq "not-helpful" }).Count -$unresolved = $total - $helpful - $notHelpful +$declined = ($data | Where-Object { $_.Verdict -eq "declined" }).Count +$incorrect = ($data | Where-Object { + $_.Replied -eq $true -and $_.Verdict -eq "not-helpful" +}).Count +$unresolved = ($data | Where-Object { + $_.Replied -eq $false -and $_.Verdict -eq "not-helpful" +}).Count $replied = ($data | Where-Object { $_.Replied -eq $true }).Count $responseRate = [math]::Round(($replied / $total) * 100, 1) +$precisionDenominator = $helpful + $incorrect +$precision = if ($precisionDenominator -gt 0) { + [math]::Round(($helpful / $precisionDenominator) * 100, 1) +} else { 0 } # Per-repo $repoStats = $data | Group-Object Repo | ForEach-Object { ... } @@ -112,10 +124,12 @@ $engStats = $data | Group-Object Engineer | ForEach-Object { ... } ## Outlook HTML Formatting Rules +> **Canonical design:** `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html` is the current **v3 newsletter** template — read it in full before generating. The primitives below (table layout, inline styles, `bgcolor`+inline pairing, bars-not-SVG) still hold; the load-bearing v3 techniques (single-row chips, edge-to-edge frame, newsletter masthead, `mso-line-height-rule`) are documented under **v3 Newsletter Rendering** below. The asset is written entirely single-quoted so it stays escaping-free (`"`/`\` count = 0) for a trivial Graph push. + Outlook strips most modern CSS. Follow these rules strictly: ### Layout -- Wrap entire body in a centered `` for consistent margins +- The current shell is a **1200px hybrid-fluid** newsletter (VML hero fallback + stacking media query), not a fixed centered table. Older reports used a centered `
`; the v3 shell supersedes it. Do not hand-roll the shell — reuse the asset's outer structure. - Use **table-based layouts only** — no flexbox, no grid, no float - All styles must be **inline** — Outlook strips ` -Copilot Code Review Effectiveness Analysis + -
+ -

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

+ +
Copilot review effectiveness - engineers adopted 78% of 138 comments; 99.1% precision, only 1 genuine error.
- -

 

Background
+ + -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

+ + + -

2. Review before merging. 38% of ignored comments were on the final commit — the engineer merged without looking at Copilot's last round. Building a habit of checking review comments before clicking merge would give this feedback a chance to have impact.

+ + -

3. Improve the review instructions. The most common dismissal reasons — "lacks domain context," "suggests tests for trivial code," "misunderstands our APIs" — are things we can address by refining our copilot-instructions.md. Adding guidance like "don't suggest tests for telemetry-only changes" or providing context about specific domain patterns could reduce the noise and make the signal-to-noise ratio better for everyone.

+
-

The Android Auth Platform team builds and maintains the authentication libraries used by Microsoft's mobile apps — including MSAL (client-side), Common (shared IPC and utilities), and Broker (the brokered authentication service running on Android devices). Our codebase spans three repositories with 10 active contributors.

+ + -

Earlier this year, we enabled GitHub Copilot code reviews across all three repositories. We also added custom review instructions via copilot-instructions.md to give the AI context about our architecture, coding conventions, and multi-repo structure — aiming to make its feedback more relevant than generic suggestions.

- -

This report answers the question: Is Copilot code review actually useful for our team? We looked at every comment Copilot left over 2 months and determined — through reply analysis, commit diff verification, and AI-assisted classification of ambiguous cases — whether each piece of feedback led to a real code improvement, was dismissed as irrelevant, or was simply never evaluated by the engineer.

+ + - -

 

At a Glance
+ +
+ + + + + + + + + + +
Window 5 weeks Comments 138 Helpful & adopted 78.3% Precision 99.1% View Org-wide (anonymized)
+
- - - - - - - - +
- - -
-
57%
-
of comments received
no response
-
-		 - - -
-
41%
-
confirmed
helpful
-
-		 - - -
-
18%
-
confirmed
not helpful
-
-		 - - -
-
41%
-
unresolved
(inconclusive)
-
-
+
What's New This Cycle
- - - - - - +
Three things that stand out this period
+
 
+
This is our fourth measurement window. Three things stand out from the data.
+
 
+
1. Engineers adopted 78.3% of Copilot's comments outright — a new high. On the code that actually shipped, only 2.2% were left unevaluated. Crucially, 23 adoptions came with no reply and were confirmed against the merged diff (13 by matching Copilot's exact suggestion to the fix commit) — so silent adoption is credited, not lost.
+
 
+
2. We measure what actually merged — plus coverage. The corpus is the 138 comments on human PRs that merged in-window, so every figure describes shipped code. Copilot reviewed 86.4% of merged human PRs (70 of 81). This “what-shipped” basis is a deliberate change from prior cycles, so the rates aren't directly comparable to earlier periods (see Trend).
+
 
+
3. Two numbers — and helpfulness leads. Helpfulness (78.3%) is the headline: how often engineers adopted a comment. Precision (99.1%) backs it up: only one of 138 comments was a genuine error. The gap between them is deliberate declines (18.8%) — correct feedback an engineer consciously chose not to take, not Copilot mistakes.
+
 
+
-The biggest finding isn't about AI quality — it's about adoption. -Of the 557 comments Copilot left, only 239 (43%) received any response from an engineer — the other 318 were never acknowledged. Of those 239 responded-to comments, 144 (60%) led to confirmed code improvements. That's a strong signal: when engineers read and evaluate Copilot's feedback, the majority of it is useful. The challenge is that 57% of comments are never evaluated at all, leaving 226 (41%) in an unresolved state where we can't determine whether they were helpful or not. -
+
At a Glance
- -

Scope

- - - - - - - - +
Helpfulness leads — precision backs it up
+
 
+
We analyzed every inline Copilot comment on the human-authored PRs that merged this period — 138 comments on shipped code.
+
 
+
- - -
-
163
-
Human PRs
scanned
-
-		 - - -
-
113
-
PRs received
Copilot review (69%)
-
-		 - - -
-
557
-
Inline review
comments
-
-		 - - -
-
4.9
-
Avg comments
per reviewed PR
-
-
+ + + + + + + +
+
+
78.3%
+
Confirmed helpful & adopted
+
108 of 138 comments engineers acted on
+
+ 		+
+
99.1%
+
Review precision
+
Only 1 of 109 judged comments was wrong
+
+
+
+
1
+
Genuine error — 0.7%
+
The only slice that reflects on the AI
+
+ 		+
+
81.2%
+
Engineer response rate
+
112 of 138 comments engaged
+
+
- - -

 

Overall Results
- -

Engineer Response Rate

-

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- - - - - - - +
 
+
43% replied57% ignored
+ +
+
How to read this
+
Two numbers, two questions. Helpfulness = did we act on it? (Helpful ÷ all comments = 108 ÷ 138 = 78.3%). Precision = when judged, was Copilot right? (Helpful ÷ (Helpful + Incorrect) = 108 ÷ 109 = 99.1%). A comment can be precise but not helpful — that's the Declined bucket, where Copilot was correct but the engineer made a deliberate call. Of comments engineers actually replied to, ~76% were adopted (85 ÷ 112); the rest were considered declines. That gap is judgment, not error.
+
+
 
+ +
Helpfulness Verdict
- - - - - - - +
The full four-way breakdown
+
 
+
Every comment lands in one of four buckets. Note the red Not helpful segment is barely visible at 0.7% — the only slice that counts against Copilot.
+
 
+
Jun 10 – Jul 15 · 138 comments
 
Engineer replied (239)
 
No response (318)
Helpful 78.3%Declined 18.8%  
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+ + +
+
Formula
+
Precision = Helpful ÷ (Helpful + Incorrect) = 108 ÷ (108 + 1) = 99.1%. Declined and Unresolved are excluded from both sides — neither is a Copilot mistake — so we never penalize the AI for feedback an engineer simply chose not to take.
+
+
 
+ +
By Repository
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- - - - - - - - +
Every repo clears 97% precision
+
 
+
41.3% Helpful18.1%40.6% Unresolved
RepositoryCommentsResponseHelpfulDeclinedIncorrectUnresolvedPrecision
Broker6978.3%52 (75.4%)15 (21.7%)0 (0.0%)2 (2.9%)100%
Common5680.4%45 (80.4%)9 (16.1%)1 (1.8%)1 (1.8%)97.8%
MSAL13100%11 (84.6%)2 (15.4%)0 (0.0%)0 (0.0%)100%
+
 
+
MSAL leads on response (100%) and helpfulness (84.6%) with zero incorrect comments. Broker carries the highest volume and the largest Declined bucket — a team that engages heavily and makes deliberate design calls — and still posts a clean 100% precision. The Incorrect column — the only one that reflects poorly on the AI — is a single comment across all three repos (in Common).
+
 
+ +
By Engineer
- - - - - - - - - +
Declined is not a failure
+
 
+
 
Confirmed Helpful (230)
 
Confirmed Not Helpful (101)
 
Unresolved (226)
EngineerCmtsRespHelpfulDeclinedIncorrectUnresPrecision
Engineer A3070%28200100%
Engineer B2282%14602100%
Engineer C2167%12801100%
Engineer D1573%13200100%
Engineer E14100%12200100%
Engineer F1182%10100100%
Engineer G8100%7100100%
Engineer H5100%401080%
Engineer I5100%3200100%
Engineer J4100%2200100%
Engineer K3100%3000100%
+
 
+ + +
+
Insight
+
The Declined column changes how we read this table. Under the old binary rubric, Engineer C (8 declined) and Engineer B (6 declined) would have looked like the engineers getting the least value from Copilot. In fact both post 100% precision — Copilot was never actually wrong on their PRs. They engage heavily and make frequent, deliberate design-tradeoff calls. Engagement and judgment, not accuracy, are what these numbers really track.
+
+
 
+ +
Trend: Historical Comparison
- - - - - - - - - - - - - - - - - - - - - - - - - - - +
This period vs previous — read across a basis change
+
 
+
One caution before the numbers: this cycle changed how we count. We scoped to merged PRs and credited silent adoption; prior periods counted all comments by creation date. So the quality metrics aren't apples-to-apples — read the pre-June rows as directional only.
+
 
+
Jun 10 – Jul 15 (current basis)
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. All 226 are comments where the engineer did not respond. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
Helpful 78.3%Declined 18.8%  
+
 
+
Apr 20 – Jun 9 (previous basis)
Helpful 52.9%16.6%Unresolved 30.6%
+
 
+ + + + + +
■ Helpful■ Declined■ Not helpful■ Unresolved
+
 
+
Current vs previous period. Two things you can read cleanly across the basis change: response rate is genuinely up, and precision — the one metric that isolates AI quality — exists from this cycle forward. Rows driven by the basis shift are marked.
+
 
+
MetricPrev · 50dThis · 35dChange
Response rate67.5%81.2%▲ +13.7pp
Replied-helpful rate75.5%75.9%≈ flat
Helpful (of all)52.9%78.3%+25.4pp · basis
Unresolved30.6%2.2%−28.4pp · basis
Copilot precisionn/a99.1%new
+
 
+
The full multi-cycle run — the “Dismissed” column combines Declined + Incorrect for continuity, and the current row (highlighted) is the only one on the merged what-shipped basis:
+
 
+
PeriodDaysCommentsCmts/wkResponseHelpfulDismissedUnresolved
Jan 24 - Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 - Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 - Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 - Jul 153513827.681.2%78.3%19.6%2.2%
+
 
+
Historical “not helpful” rates of 16–21% almost certainly overstated how often the AI was actually wrong — they bundled deliberate declines with genuine errors. Treat pre-June quality as an upper bound, not a fact. The step change at June is largely a definition change, not a behavior change.
+
 
+ +
What Copilot Is Good At
- - - - - - +
Real catches from this period
+
 
+
-226 comments (41%) are unresolved — not because the AI was wrong, but because no one engaged. -All 226 are comments that received no reply from the engineer. This includes 122 comments on the final commit before merge (where the engineer had no subsequent commits), 45 where the file was modified but at different lines, 50 where the comment had no line number for verification, and 9 where the file was never modified. If engineers had responded — even just to dismiss — we would know whether the feedback was useful. -
+
+
Copilot flagged that a flight/telemetry provider was being read before it was initialized — an ordering bug that would silently drop data. “Good catch — this is a fundamental ordering problem. Fixing it.”
+
— Broker PR #215 · AriaInitializer
+
- -

How Helpful Comments Were Delivered

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
+
+
A newly-added overload shadowed a deprecated one; Copilot recommended @JvmOverloads to preserve binary compatibility for Java callers. “Fixed. Thanks for pointing this out.”
+
— Common PR #3147
+
- -

How Not Helpful Comments Were Identified

- - - - - - - - - +
 
+
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
+
+
On an OWASP dependency check, Copilot spotted an inconsistency in how a CVE suppression was applied across modules. “Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951.”
+
— Common PR #3156
+
- - -

 

Results by Repository
- - -

Broker (293 comments)

- - - - - - +
 
+
49%16%35%
+
When Copilot Was Actually Wrong
- - -

Common (188 comments)

- - - - - - +
The one genuine error, in full — all 0.7%
+
 
+
36%18%46%
+
+
Copilot warned a Mockito varargs matcher would misbehave. “Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12.” A version-specific false positive — the engineer's repro proved Copilot wrong. This is the only genuine error in all 138 comments.
+
— Common PR #3171
+
- - -

MSAL (76 comments)

- - - - - - +
 
+
26%25%49%
+ +
+
Measurement integrity
+
The force-push confound. A prior draft nearly counted a second error: a correct Broker @Override comment (PR #212) almost recorded as a hallucination because the engineer force-pushed a rewritten commit ~2 min after opening the PR. Copilot had reviewed the original commit, where the annotation was genuinely absent; the force-push orphaned it, so GitHub marked the note “outdated.” We found this fingerprint on 31 of 138 comments (9 PRs), but only #212 was a proven wrongful-outdating — so 99.1% precision holds. Every “already there / outdated” dismissal on a force-pushed PR is now validated against the immutable reviewed snapshot before it's scored.
+
+
 
+ +
When Copilot Was Right but Declined
- - - - - - - - - - +
Correct feedback, deliberate call — neutral
+
 
+
We hand-audited all 26 Declined comments. Two things stand out: none were Copilot errors — every one was a correct or reasonable note — and all 26 were actively replied to. Each is a deliberate, reviewed call where the engineer read the note and chose otherwise.
+
 
+
 
Helpful
 
Not Helpful
 
Unresolved
+
+
Copilot flagged the comment ~1 per device as potentially misleading. “~ = approximate. i think this is ok.” A fair point the author weighed and consciously declined — a judgment call another reviewer might have taken the other way. Not a Copilot error.
+
— Broker PR #201 · the flagship case
+
- - - - - - - - - - - +
 
+
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
+
+
Copilot suggested not logging a full stack trace for an expected exception. “Since this is a new feature, we want that stack trace for all exceptions.” A sound general heuristic that simply did not apply here.
+
— Broker PR #223
+
- - -

 

Results by Engineer
-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- - - - - - - - - - - - - - - - - - - - - - +
 
+
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
+
What We'd Recommend
- - - - - +
Three opportunities
+
 
+
-Engagement drives value — and visibility. -Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
+ + + +
+
1
+ 		+
Keep engaging — it's working
+
Response rate rose to 81.2% (from 67.5%), and engineers adopted 23 comments silently. A quick reply — even a decline with a one-line reason — is what lets us separate “Copilot was wrong” from “I chose otherwise”; silent adoption we now credit automatically.
+
- - -

 

What Copilot Is Good At
- -

Catching real bugs:

-
-PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers.
-Engineer reply: "You're right. Making the change." -
- -

Stale documentation and naming inconsistencies:

-
-PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant after it was renamed. All four were silently fixed in the next commit. -
- -

CI/pipeline configuration issues:

-
-PR #3038 (Common): Copilot warned that using vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. -
- -

The @copilot apply workflow:

-
-In 16 instances (2.9%), engineers validated the feedback and delegated the fix back: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue. -
- - -

 

What Copilot Struggles With
- -

Lacking domain context:

-
-Copilot: "shared_device_id could be used for tracking — consider hashing before emission."
-Engineer: "The shared_device_id is a random UUID and is not PII." -
- -

Suggesting tests for trivial code:

-
-Copilot: "New telemetry attributes lack test coverage..."
-Engineer: "These are just telemetry related changes and adding unit tests will be overdo here." -
- -

Misunderstanding APIs:

-
-Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."
-Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." -
- - -

 

Key Takeaways
- - - - - - - - - - +
 
+
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality, the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
+ + + +
+
2
+ 		+
Validate “outdated” dismissals before scoring them
+
The force-push confound produced one near-miss this cycle: a correct comment (Broker #212) was almost recorded as a hallucination because a rebase orphaned the reviewed commit. We now validate every “already there / outdated” dismissal on a force-pushed PR against the immutable reviewed snapshot. Engineers can help by not rebasing away the reviewed commit mid-review.
+
- - -

 

How We Measured This
- - - - - - - - +
 
+
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used GitHub compare API to check if suggestion tokens appeared as diff additions, or if exact line ranges were modified.
4. AI-assisted classificationAll 133 ambiguous replies were read and classified by GitHub Copilot, based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong.
+ + + +
+
3
+ 		+
Tune copilot-instructions.md against real false-positive patterns
+
Only one genuine error slipped through (a Mockito 5.11.0 version assumption). The higher-leverage fix is the recurring declined-but-correct nitpicks: teach Broker that telemetry attribute values can be intentional literals, stop flagging approximate “~” rationale comments, and have all repos re-verify a missing @Override/test/#TBD against the PR head before flagging.
+
+
 
+ + +
How We Measured This
+
Methodology notes
+
 
+
Corpus. Every inline Copilot review comment on human-authored PRs that merged between June 10 and July 15, 2026 — 138 comments across Common, MSAL, and Broker. Bot-authored PRs and comments on unmerged or abandoned PRs are excluded, so every number describes code that actually shipped.
+
 
+
Coverage. Copilot reviewed 86.4% of merged human PRs (70 of 81). Most uncovered PRs are trivial (No-Changelog) or predate Copilot review being enabled on that repo.
+
 
+
Four-way verdict. Each comment is hand-classified into exactly one bucket: Helpful (adopted), Declined (correct or reasonable, but the engineer made a deliberate call not to take it), Incorrect (Copilot was factually wrong), or Unresolved (never evaluated). Declined and Incorrect are separated on purpose — only Incorrect counts against the AI.
+
 
+
Silent adoption. Adoption isn't read from replies alone. Where an engineer changed the code without replying, we verify against the merged diff — 13 of 23 silent adoptions were confirmed by matching Copilot's exact suggestion to the fix commit. The diff verification itself is long-standing; what changed this cycle is that we now credit these as Helpful rather than leaving them Unresolved.
+
 
+
Precision. Precision = Helpful ÷ (Helpful + Incorrect). Declined and Unresolved are excluded from both sides, so the metric answers only “when Copilot was judged, was it right?” and never penalizes the AI for feedback an engineer simply chose not to take.
+
 
+
Force-push validation. For every “already there / outdated / not relevant” dismissal we check whether the reviewed commit was later rewritten (original_commit_id vs the PR's current commit list). If it was, we re-derive Copilot's claim from the immutable diff_hunk against the merged state before scoring — so a rebase can't silently record a correct comment as an error (see Broker #212).
+
 
+
Trend caveat. Pre-June periods counted all comments by creation date under a binary rubric; this cycle scopes to merged PRs with the four-way verdict. Cross-period quality comparisons are directional only.
+
 
+
Anonymization. The org-wide edition replaces engineer names with stable Engineer A–K labels ordered by comment volume; all counts are otherwise identical to the team edition.
-

Classification rules: "Confirmed Helpful" requires positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" requires positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage are classified as "Unresolved" rather than assumed unhelpful.

- - -

 

What We'd Recommend
- -

Based on this analysis, we see three opportunities:

-

1. Engage with the feedback. The single biggest improvement would be for engineers to respond to Copilot's review comments — even if just to dismiss them. When engineers engage, 60% of comments are helpful. When they don't, 71% of comments become a black hole of unknown value. A quick "not applicable" is more useful than silence — it tells us whether the AI is giving good feedback, and it builds the data we need to improve the review instructions.

+
+ Generated by the copilot-review-analyst skill · rendered with outlook-highfidelity-report · Merged what-shipped basis (PRs merged Jun 10 – Jul 15, 2026). Methodology: four-way verdict (Helpful / Declined / Incorrect / Unresolved) + Copilot precision, with silent-adoption crediting and force-push-snapshot validation. +
+ - - -
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

- -
+ - - - - - - - - - - - - - - - - - diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html index 72759b00..0fa1f846 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.html @@ -98,314 +98,746 @@ .card-grid { grid-template-columns: repeat(2, 1fr); } .donut-container { flex-direction: column; } } + /* --- 4-way methodology additions --- */ + :root { + --blue: #0969da; --light-blue: #ddf4ff; + --purple: #8250df; --light-purple: #fbefff; + } + .card.blue { border-left: 4px solid var(--blue); } + .card.purple { border-left: 4px solid var(--purple); } + .bar-fill.blue { background: var(--blue); } + .bar-fill.purple { background: var(--purple); } + .stacked-segment.green { background: var(--green); } + .stacked-segment.blue { background: var(--blue); } + .stacked-segment.red { background: var(--red); } + .stacked-segment.amber { background: var(--amber); } + .card.six-grid { } + .card-grid.six { grid-template-columns: repeat(6, 1fr); } + @media (max-width: 640px) { .card-grid.six { grid-template-columns: repeat(2, 1fr); } }

Copilot Code Review Effectiveness Analysis

-

Android Auth Platform  |  January 23 – March 23, 2026  |  Common, MSAL, Broker Repositories

- - -

Executive Summary

- -

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories over the past two months. For each of the 557 comments, we determined whether the feedback led to a concrete code improvement.

- -
-
-
57%
-
of comments received
no response
-
-
-
41%
-
confirmed
helpful
-
-
-
18%
-
confirmed
not helpful
-
-
-
41%
-
unresolved
(not evaluated)
-
-
- -
- The biggest finding isn't about AI quality — it's about adoption. - When engineers do engage with Copilot's comments, 60% turn out to be helpful. But 57% of comments receive no response at all, and 41% of all comments are unresolved — we can't tell if they were useful because no one evaluated them. The true helpfulness rate lies between 41% (confirmed floor) and 82% (if all unresolved were helpful). -
- - -

Scope

- -
-
-
163
-
Human PRs
scanned
-
-
-
113
-
PRs received
Copilot review (69%)
-
-
-
557
-
Inline review
comments
-
-
-
4.9
-
Avg comments
per reviewed PR
-
+

Android Auth Platform  |  June 10 – July 15, 2026  |  Common, MSAL, Broker Repositories

+ +

Executive Summary

+

We analyzed every inline code review comment left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the 132 comments, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because Copilot was wrong or because the engineer made a deliberate judgment call.

+
+
72%
engineer
response rate
+
54.5%
confirmed
helpful
+
20.5%
correct but
declined
+
1.5%
actually
incorrect
+
23.5%
unresolved
(no evidence)
+
97.3%
review
precision
- - -

Overall Results

- -

Engineer Response Rate

+
Fairness principleWe separate “Declined” (Copilot was correct or reasonable, but the engineer intentionally chose not to act — a nitpick, a stylistic preference, or an approximation the author knew about) from “Incorrect” (Copilot was factually wrong). Only genuinely Incorrect comments count against review quality. Precision = Helpful ÷ (Helpful + Incorrect), excluding Declined and Unresolved — so we never penalize Copilot for feedback it had no way to know the author would decline.
+

This run introduces a new, fairer methodology. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral Declined category and reserves Not Helpful exclusively for comments where Copilot was demonstrably wrong.

+

Key findings:

+
    +
  • 72% of Copilot's comments now receive a response from engineers. This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value.
    • +
  • Only 1.5% of all comments were genuinely wrong. Across 132 comments, just two were confirmed as Copilot errors. When you compute Copilot's precision — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at 97.3%. The AI is rarely incorrect.
    • +
  • 20.5% of comments were correct-but-declined. These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. These do not count against Copilot's quality. Treating them as failures, as the old methodology did, materially understated the tool's accuracy.
    • +
  • 54.5% of all comments led to a confirmed code improvement. With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy.
    • +
+
+

How We Measured This

+

This analysis went through five phases to ensure accuracy:

+
    +

  1. Data collection. We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text.

    +
    2. +

  2. Diff-level verification. For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit.

    +
    3. +

  3. AI-assisted reply classification (three-way). For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts:

    +
      +
    • Helpful — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot).
      • +
    • Declined — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). Copilot was not wrong — it simply had no way to know the author's intent.
      • +
    • Not Helpful (incorrect) — the engineer's reply demonstrates the comment was factually or technically wrong (a false positive, a hallucinated problem, or a misunderstanding of a library/API).
      • +
    +
    4. +

  4. The fairness principle. The dividing line between Declined and Not Helpful is whether Copilot was wrong, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is Declined, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as Not Helpful.

    +
    5. +

  5. Cross-validation. All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved.

    +
    6. +
+

The final dataset classifies each of the 132 comments as confirmed helpful, declined (correct but not adopted), not helpful (incorrect), or unresolved (insufficient evidence to determine). No comment is left without a classification.

+
+

A Note on Fairness: Why "Declined" Is Not "Wrong"

+

The single most important change in this report is the recognition that an engineer declining a comment is not evidence that the comment was bad.

+

Consider Broker PR #201. Copilot flagged a code comment that read ~1 per device as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: "~ = approximate. i think this is ok." Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair.

+

We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise.

+

By separating these Declined cases (20.5% of all comments) from genuine errors (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually wrong in only 2 of 132 comments.

+
+

Overall Results

+ + + + + + + + + + + + + + + + + + + + + + + + + +
MetricValue
Human PRs scanned87
PRs that received Copilot review53 (61%)
Total inline review comments132
Average comments per reviewed PR2.5
+

Engineer Response Rate

Before looking at helpfulness, it's important to understand how engineers interact with Copilot reviews — because a comment can only demonstrate value if someone reads it.

- -
-
43% replied
-
57% ignored
-
-
- Engineer replied (239) - No response (318) -
- -

More than half of all Copilot review comments receive no human response. The majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it.

- -

Helpfulness Verdict

- + + + + + + + + + + + + + + + + + + + + +
BehaviorCountPercentage
Engineer replied (any response — acceptance, decline, or discussion)9572.0%
Engineer did not reply3728.0%
+

At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a 100% response rate across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted.

+

Helpfulness Verdict

+

Each comment was classified into one of four categories. Note the deliberate split between Declined (Copilot was right, the engineer chose otherwise) and Not Helpful (Copilot was wrong):

-
41.3%
-
18.1%
-
40.6%
+
54.5%
+
20.5%
+
+
23.5%
- Confirmed Helpful (230) - Confirmed Not Helpful (101) - Unresolved (226) + Helpful (72) + Declined (27) + Incorrect (2) + Unresolved (31)
- - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
VerdictCount%Definition
Confirmed Helpful23041.3%The comment led to a code change — engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Confirmed Not Helpful10118.1%The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design.
Unresolved22640.6%No reply and no definitive diff evidence. Includes comments on the final commit before merge, files modified at different lines, and comments with no line number. Could be valid feedback that was never evaluated.
VerdictCountPercentageDefinition
Confirmed Helpful7254.5%The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff.
Declined (correct, not adopted)2720.5%The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. This is neutral; Copilot was not wrong.
Not Helpful (incorrect)21.5%The engineer demonstrated the comment was factually or technically wrong — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality.
Unresolved3123.5%The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated.
- -
- 226 comments (41%) are unresolved — not because the AI was wrong, but because no one looked. - This includes 122 comments on the final commit before merge where the engineer had no subsequent commits, 45 where the file was modified at different lines, 50 with no line number for verification, and 9 where the file was never modified. If engineers had engaged — even just to dismiss — we would know. -
- -

How Helpful Comments Were Delivered

+

The Precision Metric

+

Because Declined is neutral and Unresolved is unknown, the fairest single measure of Copilot's review quality is its precision — of the comments we can actually adjudicate as right or wrong, what fraction were right?

+
+

Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%

+
+

Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. 97.3% precision means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy.

+

How Each Category Breaks Down

+

Confirmed Helpful (72):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and acknowledged144"good catch", "fixed", "addressed", "added unit test", "@copilot apply changes"
Engineer silently applied the fix86No reply, but suggestion code or exact line range verified as modified in subsequent commit
Total Confirmed Helpful230
PathCountDescription
Engineer replied and acknowledged66Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …")
Engineer silently applied the fix6No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit
- -

How Not Helpful Comments Were Identified

+

Declined — correct but not adopted (27): All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong.

+

Not Helpful — incorrect (2):

- - - - + + + + + + + + + + + + + + + + + + +
PathCountDescription
Engineer replied and dismissed95"won't fix", "this is fine", "not applicable", "Copilot is incorrect"
Comment on stale/outdated code6Comment was on code already changed in a different commit
Total Confirmed Not Helpful101
CommentRepo / PRWhy it was wrong
Mockito varargs matcher warningCommon #3171Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12.
"Missing @Override annotation"Broker #212The @Override annotation was already present — a hallucinated problem.
- - -

Results by Repository

- -
-
-
Broker
-
-
48.5%
-
16%
-
35%
-
-
293
-
-
-
Common
-
-
36.2%
-
18%
-
46%
-
-
188
-
-
-
MSAL
-
-
26%
-
25%
-
49%
-
-
76
-
-
-
- Helpful - Not Helpful - Unresolved - (number = total comments) -
- +

Unresolved (31): No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated.

+
+

Results by Repository

- - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RepositoryCommentsResponse RateHelpfulNot HelpfulUnresolved
Broker29356.0%142 (48.5%)48 (16.4%)103 (35.2%)
Common18829.8%68 (36.2%)34 (18.1%)86 (45.7%)
MSAL7625.0%20 (26.3%)19 (25.0%)37 (48.7%)
RepositoryCommentsResponse RateHelpfulDeclinedNot HelpfulUnresolvedPrecision
Broker6669.7%33 (50.0%)16 (24.2%)1 (1.5%)16 (24.2%)97.1%
Common5168.6%29 (56.9%)7 (13.7%)1 (2.0%)14 (27.5%)96.7%
MSAL1593.3%10 (66.7%)4 (26.7%)0 (0.0%)1 (6.7%)100%
- -

Broker has the highest response rate (56%) and the highest confirmed helpfulness (49%). In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved.

- - -

Results by Engineer

-

Each engineer has two GitHub accounts (personal + EMU). These have been merged. Names are anonymized.

- +

Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository.

+

Coverage across the three repos:

- - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EngineerCommentsRepliedResponse RateHelpfulNot HelpfulUnresolvedHelpfulness
Engineer A2020100%146070.0%
Engineer B837590.4%5726068.7%
Engineer C15426.7%82553.3%
Engineer D403075.0%1914747.5%
Engineer E1103733.6%44184840.0%
Engineer F992020.2%36115236.4%
Engineer G632234.9%20123131.7%
Engineer H1002424.0%27126127.0%
Engineer I24625.0%501920.8%
Engineer J3133.3%0030%
RepositoryHuman PRsPRs Reviewed by CopilotCoverage
Common382361%
MSAL14750%
Broker352366%
- -
- Engagement drives value — and visibility. - Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness and zero unresolved comments. When you engage, you know exactly what the AI got right and wrong. Engineer F (20%) and Engineer H (24%) have 52 and 61 unresolved comments respectively — over half their feedback goes into a black hole. -
- - -

Response Behavior Deep Dive

- -

What happens when engineers reply? (239 comments)

-
-
60% helpful
-
40% not helpful
-
-

When engineers engage, the majority of Copilot feedback turns out useful. This is the strongest signal that review quality is decent — the bottleneck is adoption.

- -

What happens when engineers don't reply? (318 comments)

-
-
27% silently fixed
-
-
71% unresolved
-
- +
+

Results by Engineer

+

Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break.

- - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code silently applied (verified via diff)5015.7%Confirmed Helpful
Exact commented lines modified in subsequent commit72.2%Confirmed Helpful
Evidence of fix at nearby lines (re-audit)299.1%Confirmed Helpful
Merged without any subsequent commits12238.4%Unresolved
File modified but not at the commented lines4514.2%Unresolved
File modified but no line number to verify5015.7%Unresolved
File never modified after the comment92.8%Unresolved
Comment on stale/outdated code61.9%Confirmed Not Helpful
EngineerCommentsRepliedResponse RateHelpfulDeclinedNot HelpfulUnresolvedHelpfulnessPrecision
Engineer A77100%7000100%100%
Engineer B33100%3000100%100%
Engineer C11981.8%900281.8%100%
Engineer D6583.3%401166.7%80.0%
Engineer E302376.7%1770656.7%100%
Engineer F9555.6%520255.6%100%
Engineer G11654.5%601454.5%85.7%
Engineer H6350.0%300350.0%100%
Engineer I44100%220050.0%100%
Engineer J191578.9%890242.1%100%
Engineer K191578.9%870442.1%100%
Engineer L300%00030%n/a
Engineer M300%00030%n/a
Engineer N100%00010%n/a
- -

The single largest category is the 122 comments (38% of ignored) where the engineer merged the PR without pushing any additional commits after Copilot's review. These comments may have been valid, but we cannot tell because the engineer never evaluated them.

- - -

What Copilot Is Good At

- -

Catching real bugs:

+

Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.

+

Key observation — the Declined column changes how we read this table. Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the least value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a 100% precision: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket.

+
+

Response Behavior Deep Dive

+

Of the 132 total comments:

+
    +
  • 95 (72.0%) received a reply. Of those, 69.5% were helpful, 28.4% were declined (correct but not adopted), and only 2.1% were incorrect. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong.
    • +
  • 37 (28.0%) were not replied to. Of those, 16.2% were silently addressed (verified via diff) and the remaining 83.8% are unresolved — we cannot determine whether the comment was useful because the engineer never evaluated it.
    • +
+

What happens to comments engineers reply to

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Reply verdictCount% of repliedCounts against Copilot?
Helpful — accepted or fixed6669.5%No — positive
Declined — correct but not adopted2728.4%No — neutral
Not Helpful — incorrect22.1%Yes
+

The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot wrong only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently.

+

What happens to ignored comments

+ + + + + + + + + + + + + + + + + + + + + + + +
What happenedCount% of ignoredVerdict
Suggestion code / exact lines silently applied (verified via diff)616.2%Confirmed Helpful
No reply and no diff evidence either way3183.8%Unresolved
+

The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further.

+
+

What Copilot Is Good At

+

The most valuable helpful comments this period, with real examples from our PRs:

+

Catching real initialization-ordering bugs:

-

PR #3050 (Common): Copilot flagged that "$it" string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters.

-

Engineer reply: "You're right. Making the change."

+

PR #215 (Broker) — AriaInitializer: Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."

+

This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone.

- -

Stale documentation and naming inconsistencies:

+

Telemetry schema and contract discrepancies:

-

PR #64 (Broker): Copilot identified four locations where KDoc still referenced the old flight constant USE_TEE_ONLY_FOR_TOKEN_BINDING after it was renamed. All four were silently fixed in the next commit.

+

PR #204 (Broker) — AccountChooser: Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +Engineer acknowledged and reconciled the schema.

- -

Dead code and unused imports:

+

Deprecated-API and compatibility issues:

-

PR #3040 (Common): Copilot spotted an unused local variable enabledSettingRaw. Verified as fixed via diff — the suggested replacement code appeared in the commit additions.

+

PR #3147 (Common): Copilot pointed out a newly-added overload shadowed a deprecated one and recommended @JvmOverloads / a default argument to preserve binary compatibility for Java callers. +Engineer reply: "Fixed. Thanks for pointing this out."

- -

CI/pipeline configuration issues:

+

Security and policy consistency:

-

PR #3038 (Common): Copilot warned that vmImage: 'windows-latest' makes the CD pipeline non-deterministic. The engineer changed to a pinned image version.

+

PR #3156 (Common) — OWASP dependency check: Copilot identified an inconsistency in how a CVE suppression was applied across modules. +Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."

+

Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item.

- -

The @copilot apply workflow:

+

Behavior-vs-description correctness:

-

In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back with: @copilot open a new pull request to apply changes based on [this feedback]. An efficient pattern where AI identifies and fixes the issue.

+

PR #3165 (Common) — Authority validation: Copilot noted a URL comparison was matching on the full URL when only the host should matter. +Engineer reply: "Good point — updated the code to check host only."

- - -

What Copilot Struggles With

- -

Lacking domain context:

+
+

When Copilot Was Actually Wrong

+

Only two comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule:

+

Misjudging a library's behavior:

-

Copilot: "shared_device_id could be used for tracking across apps — consider hashing before emission."

-

Engineer: "The shared_device_id is a random UUID generated by one of the participating apps and is not PII."

+

PR #3171 (Common): Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."

+

Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive.

- -

Suggesting tests for trivial code:

+

Hallucinating a missing annotation:

-

Copilot: "New telemetry attributes lack test coverage..."

-

Engineer: "These are just telemetry related changes and adding unit tests will be overdo here."

+

PR #212 (Broker): Copilot claimed a method was missing an @Override annotation. +Engineer reply: "The override annotation is already there."

+

The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero.

- -

Misunderstanding APIs:

+
+

When Copilot Was Right but the Engineer Declined

+

These comments were not wrong — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction.

+

Deliberate notation choice (the flagship case):

-

Copilot: "00000003-0000-0ff1-ce00-000000000000 is the resource ID for Microsoft Graph, not SharePoint Online."

-

Engineer: "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online."

+

PR #201 (Broker): Copilot flagged the code comment ~1 per device as potentially misleading, since the tilde could be read as an exact value. +Engineer reply: "~ = approximate. i think this is ok."

+

A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure.

- -

Commenting on intentional design choices:

+

Subjective naming preference:

-

Copilot: "getPackageInfo() != null is redundant since it either returns PackageInfo or throws..."

-

Engineer: "This is fine. The verbosity makes the code clearer to understand."

+

PR #3151 (Common): Copilot suggested encoding the "prompt" semantics into a span name for clarity. +Engineer reply: "I think the current name is fine."

+

A judgment call on naming. Both names are defensible; the author preferred the existing one.

- - -

Key Takeaways

- - - - - - - - - - - -
#Finding
157% of Copilot review comments receive no response from engineers. The majority of AI review feedback is never acknowledged. Of the ignored comments, only 27% are silently addressed — the remaining 71% are unresolved.
241% of all comments led to a confirmed code improvement. But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement.
3When engineers engage, 60% of comments are helpful. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy.
4Only 18% of comments are confirmed not helpful. When we restrict "not helpful" to comments with actual evidence of poor quality — explicit engineer dismissals — the rate is surprisingly low.
5Engagement is the strongest predictor of value. Engineers who reply to 75%+ of comments see 47-70% helpfulness with zero unresolved. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets.
638% of ignored comments are on the final commit before merge. These 122 comments represent the last review round being skipped — the feedback had zero chance of impact regardless of its quality.
7Broker gets the most value (49% helpful), Common is middling (36%), MSAL is lowest (26%). This correlates with response rate: Broker 56%, Common 30%, MSAL 25%.
8At ~5 comments per PR, the signal-to-noise question can't be settled without engagement. We confirmed ~2 useful comments per PR on average, but with 41% unresolved, the actual number could be higher.
- - -

How We Measured This

- +

Intentional verbosity for a new feature:

+
+

PR #223 (Broker): Copilot suggested not logging a full stack trace for an expected exception type. +Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."

+

The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here.

+
+

Context Copilot couldn't have known:

+
+

PR #3177 (Common): Copilot suggested removing an INFO-level log as noise. +Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."

+

The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff.

+
+
+

Trend Over Time

+

This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the Declined vs. Incorrect split and the Precision metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table.

- - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PhaseMethod
1. Data collectionGitHub API extraction of all 557 Copilot inline review comments from 163 human-authored PRs. Excluded PRs by Copilot coding agent.
2. Reply classificationAutomated keyword matching on 239 replied comments to classify as positive, negative, or ambiguous.
3. Diff verificationFor 318 unreplied comments, used the GitHub compare API to check if suggestion tokens appeared as diff additions, or if the exact line range was modified in subsequent commits.
4. AI-assisted classificationAll 133 ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on reply text and domain context. These classifications were reviewed by the report author but not independently verified by the original PR engineers.
5. Cross-validationAll initially "not helpful" classifications were re-examined against diff evidence. 18 were reclassified where evidence was strong (e.g., typo fix + matching -1 line file change).
PeriodDaysCommentsComments/wkResponse RateHelpfulDismissed (combined)Unresolved
Jan 24 – Mar 256057066.344.4%38.6%17.4%44.0%
Mar 24 – Apr 192617045.978.8%62.9%21.2%15.9%
Apr 20 – Jun 95015722.167.5%52.9%16.6%30.6%
Jun 10 – Jul 153513226.472.0%51.5%22.0%26.5%
+

What changed this period:

+
    +
  • Response rate rose from 67.5% to 72.0%, driven partly by engineers going back to reply to previously-unanswered comments.
    • +
  • Trend-series helpfulness dipped slightly (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable.
    • +
  • The "Dismissed" number is misleading in isolation. This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that 20.5 of those 22.0 points are Declined (Copilot was right) and only 1.5 points are genuine errors. Copilot's precision this period is 97.3%. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong.
    • +
+

Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.

+

Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.

+
+

Key Takeaways

+
    +

  1. Copilot is almost never wrong — precision is 97.3%. Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74.

    +
    2. +

  2. "Declined" is not "wrong," and it's now counted fairly. 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent.

    +
    3. +

  3. Response rate climbed to 72%. Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments.

    +
    4. +

  4. 54.5% of comments led to a confirmed improvement. With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated.

    +
    5. +

  5. Every repository posts 96%+ precision. MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo.

    +
    6. +

  6. The Declined bucket concentrates in high-engagement engineers. Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem.

    +
    7. +

  7. Unresolved is the only real blind spot. 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report.

    +
    8. +

  8. The bottleneck is adoption and engagement, not accuracy. The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback.

    +
    9. +
+
+

What We'd Recommend

+

Based on this analysis, we see three opportunities:

+

1. Keep engaging — it's working. Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement.

+

2. Close the unresolved gap before merge. The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact.

+

3. Target the two real error patterns. The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining copilot-instructions.md — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero.

+
+

Methodology Notes

+
    +
  • Scope. Only inline code review comments from the Copilot user were counted. PR-level summary comments were excluded from the helpfulness analysis.
    • +
  • Bot exclusions. PRs authored by copilot-swe-agent (Copilot coding agent), dependabot[bot], and github-actions[bot] were excluded. Only PRs authored by human engineers were analyzed.
    • +
  • Three-way reply classification. Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is whether Copilot was factually/technically wrong, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers.
    • +
  • Precision. Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake.
    • +
  • Diff verification. For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as + lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed.
    • +
  • Conservative "Unresolved." Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful."
    • +
  • Account merging. Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings.
    • +
  • Data availability. Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at %TEMP%\copilot-review-analysis\ for independent verification.
    • +
+
+

Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.

-

Classification rules: A comment is "Confirmed Helpful" only with positive evidence (engineer acknowledgment or verified code change). "Confirmed Not Helpful" only with positive evidence of poor quality (explicit engineer dismissal). Comments where the engineer did not engage — including the final review round merged without evaluation — are classified as "Unresolved" rather than assumed unhelpful.

- -
-
-

Analysis conducted March 23–24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.

-

Raw data (557 comments with full text, replies, diff evidence, and verdicts) available for independent verification.

-
- + \ No newline at end of file diff --git a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md index e0764b36..d4c6dda0 100644 --- a/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md +++ b/.github/skills/copilot-review-analyst/assets/Copilot-Code-Review-Effectiveness-Report.md @@ -1,19 +1,21 @@ # Copilot Code Review Effectiveness Analysis -**Android Auth Platform | January 23 – March 23, 2026** +**Android Auth Platform | June 10 – July 15, 2026** --- ## Executive Summary -We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past two months. For each of the **557 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs. +We analyzed **every inline code review comment** left by GitHub Copilot on human-authored pull requests across our three Android Auth repositories (Common, MSAL, Broker) over the past five weeks. For each of the **132 comments**, we determined whether the feedback led to a concrete code improvement — either through an explicit engineer response, or by verifying that the suggested change appeared in subsequent commit diffs — and, critically, whether any dismissal was because *Copilot was wrong* or because *the engineer made a deliberate judgment call*. + +This run introduces a **new, fairer methodology**. In prior reports, any comment an engineer declined was lumped into a single "not helpful" bucket. That was unfair to Copilot: a comment can be entirely correct and well-reasoned, yet the author still decides not to act on it because of context the AI had no way to know (a deliberate design choice, a subjective naming preference, or a readability nit). This report separates those cases into a neutral **Declined** category and reserves **Not Helpful** exclusively for comments where Copilot was demonstrably *wrong*. **Key findings:** -- **57% of Copilot's comments received no response from engineers.** This is the single most important number in this report. The majority of AI review feedback is never even acknowledged. -- **Of comments that engineers engaged with, 60% were helpful** — a strong signal that the review quality itself is decent. -- **41% of all comments led to a confirmed code improvement.** But 41% are unresolved — comments that were ignored and where we lack evidence to judge either way. The true helpfulness rate could be significantly higher, but we can't confirm it because no one looked. -- **Engineers who reply to 75%+ of comments see 47-70% helpfulness.** Engineers who reply to <35% see 20-40%. The biggest lever for improving Copilot review value is not the AI — it's engineer engagement with the feedback. +- **72% of Copilot's comments now receive a response from engineers.** This is up sharply from prior periods (44% → 79% → 68% → 72%) and reflects a real behavior change — several engineers went back and replied to comments they had previously left unanswered. When engineers engage, we can actually judge the AI's value. +- **Only 1.5% of all comments were genuinely wrong.** Across 132 comments, just **two** were confirmed as Copilot errors. When you compute Copilot's **precision** — helpful comments as a share of everything we can actually adjudicate (helpful + wrong) — it lands at **97.3%**. The AI is rarely incorrect. +- **20.5% of comments were correct-but-declined.** These 27 comments were reasonable feedback the engineer chose not to adopt for a stated reason — by design, a subjective preference, or context Copilot couldn't have known. **These do not count against Copilot's quality.** Treating them as failures, as the old methodology did, materially understated the tool's accuracy. +- **54.5% of all comments led to a confirmed code improvement.** With only 1.5% confirmed wrong and 23.5% unresolved (never evaluated), the true helpfulness ceiling is very high — the gap is engineer engagement, not AI accuracy. --- @@ -21,17 +23,32 @@ We analyzed **every inline code review comment** left by GitHub Copilot on human This analysis went through five phases to ensure accuracy: -1. **Data collection.** We used the GitHub API to extract all 557 Copilot inline review comments from 163 human-authored PRs (excluding PRs authored by Copilot coding agent). We also recorded which comments received human replies and what those replies said. +1. **Data collection.** We used the GitHub API to extract all 132 Copilot inline review comments from 53 reviewed human-authored PRs (excluding PRs authored by the Copilot coding agent). We also recorded which comments received human replies and captured the full reply text. -2. **Reply-based classification.** For the 239 comments (43%) that received a human reply, we classified the reply as positive (e.g., "good catch", "fixed", "addressed"), negative (e.g., "won't fix", "not applicable", "by design"), or ambiguous. +2. **Diff-level verification.** For the 37 comments (28%) that received no reply, we checked whether the engineer silently acted on the feedback. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. -3. **Diff-level verification.** For the 318 comments (57%) that received no reply, we checked whether the engineer acted on the feedback silently. We used the GitHub compare API to examine the diff between the commit Copilot reviewed and the final PR head. For comments containing GitHub suggestion blocks, we checked if the suggested code tokens appeared as additions in the diff. For prose comments, we checked if the exact line range was modified in a subsequent commit. +3. **AI-assisted reply classification (three-way).** For the 95 comments (72%) that received a human reply, the AI conducting this analysis read the full Copilot comment and the engineer's reply in context, and assigned one of three verdicts: + - **Helpful** — the engineer accepted the feedback (acknowledged it, fixed it, or delegated the fix back to Copilot). + - **Declined** — the engineer's reply shows the feedback was understood and reasonable, but they made a deliberate choice not to act on it (by design, subjective preference, moot/out-of-scope, or context Copilot couldn't have known). **Copilot was not wrong** — it simply had no way to know the author's intent. + - **Not Helpful (incorrect)** — the engineer's reply demonstrates the comment was factually or technically *wrong* (a false positive, a hallucinated problem, or a misunderstanding of a library/API). -4. **AI-assisted reply classification.** All 133 comments with ambiguous replies were individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. For example, "this is just telemetry" was classified as dismissed, "@copilot apply changes" was classified as helpful (engineer delegated the fix back to Copilot), and "Added unit test for this" was classified as helpful (engineer acted on the suggestion). These classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. +4. **The fairness principle.** The dividing line between *Declined* and *Not Helpful* is **whether Copilot was wrong**, not whether the engineer acted. If an engineer says "good point, but we want the verbose stack trace for this new feature," Copilot's observation was legitimate — the author just weighed the tradeoff differently. That is **Declined**, and it is neutral. Only when the engineer shows the comment was actually incorrect ("the annotation is already there," "not reproducible on this version") does it count as **Not Helpful**. -5. **Cross-validation.** All comments initially classified as "not helpful" were re-examined against the diff evidence. 18 were reclassified as helpful where the evidence was strong (e.g., a typo fix suggestion with a corresponding -1 line file change, or an unused import suggestion with a matching -2 line change). +5. **Cross-validation.** All no-reply comments were re-examined against the diff evidence. Where the evidence of a silent fix was strong (the suggested tokens or exact lines appeared in a later commit), the comment was reclassified as Helpful rather than left Unresolved. -The final dataset classifies each of the 557 comments as **confirmed helpful**, **confirmed not helpful**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. +The final dataset classifies each of the 132 comments as **confirmed helpful**, **declined (correct but not adopted)**, **not helpful (incorrect)**, or **unresolved** (insufficient evidence to determine). No comment is left without a classification. + +--- + +## A Note on Fairness: Why "Declined" Is Not "Wrong" + +The single most important change in this report is the recognition that **an engineer declining a comment is not evidence that the comment was bad.** + +Consider Broker PR #201. Copilot flagged a code comment that read `~1 per device` as potentially misleading — the tilde could be read as an exact value. The engineer replied simply: *"~ = approximate. i think this is ok."* Copilot's observation was perfectly reasonable — an ambiguous notation is worth a second look. The author simply judged that, in context, the notation was clear enough. Copilot had no way to know the author's tolerance for that ambiguity beforehand. Counting this against the AI's review quality would be unfair. + +We saw this pattern repeatedly: Copilot suggests removing a log line, and the engineer explains it's deliberately there to catch a regression in a specific scenario; Copilot proposes a different span name, and the engineer prefers the existing one; Copilot flags a full stack-trace log, and the engineer wants exactly that verbosity for a brand-new feature. In every case the AI's reasoning was sound. The engineer had additional context — and the right — to decide otherwise. + +By separating these **Declined** cases (20.5% of all comments) from genuine **errors** (1.5%), we get an honest picture: Copilot's feedback is correct or reasonable the overwhelming majority of the time. The old binary rubric — which would have reported ~22% "not helpful" this period — obscured that the AI was actually *wrong* in only 2 of 132 comments. --- @@ -39,11 +56,10 @@ The final dataset classifies each of the 557 comments as **confirmed helpful**, | Metric | Value | |--------|-------| -| Human PRs scanned | 163 | -| PRs that received Copilot review | 113 (69%) | -| Total inline review comments | 557 | -| PR-level summary comments | 205 | -| Average comments per reviewed PR | 4.9 | +| Human PRs scanned | 87 | +| PRs that received Copilot review | 53 (61%) | +| Total inline review comments | 132 | +| Average comments per reviewed PR | 2.5 | ### Engineer Response Rate @@ -51,220 +67,272 @@ Before looking at helpfulness, it's important to understand how engineers intera | Behavior | Count | Percentage | |----------|-------|------------| -| **Engineer replied** (any response — acceptance, dismissal, or discussion) | 239 | **42.9%** | -| **Engineer did not reply** | 318 | **57.1%** | +| **Engineer replied** (any response — acceptance, decline, or discussion) | 95 | **72.0%** | +| **Engineer did not reply** | 37 | **28.0%** | -More than half of all Copilot review comments receive no human response at all. This means the majority of AI feedback enters a void — it may be valid, but we can never confirm its value if no one engages with it. +At 72%, this is the second-highest response rate we've recorded, and a marked improvement over the first analysis period (44%). Several engineers returned to previously-ignored comments and replied — most notably one engineer who went from a mix of ignored/unanswered comments to a **100% response rate** across all seven of their comments. Every reply, even a decline, is valuable: it lets us tell the difference between feedback that was wrong and feedback that was simply not adopted. ### Helpfulness Verdict -Each comment was classified into one of three categories: +Each comment was classified into one of four categories. Note the deliberate split between **Declined** (Copilot was right, the engineer chose otherwise) and **Not Helpful** (Copilot was wrong): | Verdict | Count | Percentage | Definition | |---------|-------|------------|------------| -| **Confirmed Helpful** | **230** | **41.3%** | The comment led to a code change — either the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | -| **Confirmed Not Helpful** | **101** | **18.1%** | The engineer explicitly dismissed the comment with a reason why the feedback was incorrect, irrelevant, or by design. We have positive evidence that the comment was *wrong*, not merely that it was *ignored*. | -| **Unresolved** | **226** | **40.6%** | The comment received no reply AND we could not confirm whether it was addressed. This includes cases where the engineer merged without any subsequent commits (the final review round was never evaluated), where the file was modified but not at the specific lines Copilot flagged, or where the comment had no line number making verification impossible. | +| **Confirmed Helpful** | **72** | **54.5%** | The comment led to a code change — the engineer explicitly acknowledged it, or the suggested fix was verified in a subsequent commit diff. | +| **Declined (correct, not adopted)** | **27** | **20.5%** | The engineer understood the feedback and made a deliberate choice not to act on it — by design, a subjective preference, moot/out-of-scope, or context Copilot couldn't have known. **This is neutral; Copilot was not wrong.** | +| **Not Helpful (incorrect)** | **2** | **1.5%** | The engineer demonstrated the comment was factually or technically *wrong* — a false positive, a hallucination, or a misunderstanding of a library/API. This is the only bucket that counts against Copilot's quality. | +| **Unresolved** | **31** | **23.5%** | The comment received no reply AND we could not confirm whether it was addressed. Could be valid feedback that was never evaluated. | + +### The Precision Metric + +Because *Declined* is neutral and *Unresolved* is unknown, the fairest single measure of Copilot's review quality is its **precision** — of the comments we can actually adjudicate as right or wrong, what fraction were right? + +> **Precision = Helpful ÷ (Helpful + Incorrect) = 72 ÷ (72 + 2) = 97.3%** -The **unresolved** category is the most important number in this report. These 226 comments — **41% of all feedback** — are not confirmed failures of the AI. They are comments where we simply lack evidence either way because no one engaged with them. Many may be perfectly valid feedback that was never read. If engineers had engaged with them (even just to dismiss), we would know. The true helpfulness rate likely falls somewhere between 41% (confirmed floor) and 82% (if all unresolved were helpful). +Declined and Unresolved comments are excluded from both the numerator and the denominator, because neither represents a Copilot mistake. **97.3% precision** means that when an engineer engaged deeply enough for us to judge correctness, Copilot's comment was right 72 times out of 74. The AI is almost never wrong; the open question is adoption and engagement, not accuracy. ### How Each Category Breaks Down -**Confirmed Helpful (230):** +**Confirmed Helpful (72):** | Path | Count | Description | |------|-------|-------------| -| Engineer replied and acknowledged | 144 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "addressed", "added unit test") | -| Engineer silently applied the fix | 86 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | +| Engineer replied and acknowledged | 66 | Engineer explicitly confirmed the feedback was useful (e.g., "good catch", "fixed", "done", "implemented in commit …") | +| Engineer silently applied the fix | 6 | No reply, but the suggestion code or exact line range was verified as modified in a subsequent commit | -**Confirmed Not Helpful (101):** +**Declined — correct but not adopted (27):** All 27 received an engineer reply. In every case the engineer engaged with the feedback and gave a reason for not acting on it — a design decision, a subjective preference, a moot/out-of-scope note, or domain context Copilot couldn't have known. None indicate the comment was wrong. -| Path | Count | Description | -|------|-------|-------------| -| Engineer replied and dismissed | 95 | Engineer explicitly explained why the comment was wrong, irrelevant, or by design (e.g., "won't fix", "this is fine", "Copilot is incorrect", "false positive") | -| Comment on code already changed | 6 | Comment was on stale/outdated code that had already been modified in a different commit | +**Not Helpful — incorrect (2):** -**Unresolved (226):** +| Comment | Repo / PR | Why it was wrong | +|---------|-----------|------------------| +| Mockito varargs matcher warning | Common #3171 | Engineer verified the flagged behavior does not reproduce on Mockito 5.11.0; the full suite passed 12/12. | +| "Missing `@Override` annotation" | Broker #212 | The `@Override` annotation was already present — a hallucinated problem. | -| Path | Count | Description | -|------|-------|-------------| -| Merged without any subsequent commits | 122 | No commits after Copilot's review — the engineer merged the PR without acting on the final review round. The comment may have been valid, but we cannot tell because the engineer never evaluated it. | -| File modified but not at the commented lines | 45 | The file was changed after the review, but the diff shows the changes were at different lines than what Copilot flagged. Possibly addressed via a different approach, or possibly coincidental. | -| File modified but no line number to verify | 50 | Copilot's comment had no line number metadata, and the file was modified. We cannot confirm whether the specific concern was addressed. | -| File never modified after the comment | 9 | The file was not touched in any commit after the review, and no reply was left. The comment may have been valid but was ignored. | +**Unresolved (31):** No reply and no diff evidence either way. These are not confirmed failures — they are comments no one evaluated. --- ## Results by Repository -| Repository | Comments | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | -|------------|----------|---------------|------|------|------| -| **Broker** | 293 | **56.0%** | 142 (48.5%) | 48 (16.4%) | 103 (35.2%) | -| **Common** | 188 | **29.8%** | 68 (36.2%) | 34 (18.1%) | 86 (45.7%) | -| **MSAL** | 76 | **25.0%** | 20 (26.3%) | 19 (25.0%) | 37 (48.7%) | +| Repository | Comments | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Precision | +|------------|----------|---------------|---------|----------|-------------|------------|-----------| +| **Broker** | 66 | **69.7%** | 33 (50.0%) | 16 (24.2%) | 1 (1.5%) | 16 (24.2%) | **97.1%** | +| **Common** | 51 | **68.6%** | 29 (56.9%) | 7 (13.7%) | 1 (2.0%) | 14 (27.5%) | **96.7%** | +| **MSAL** | 15 | **93.3%** | 10 (66.7%) | 4 (26.7%) | 0 (0.0%) | 1 (6.7%) | **100%** | -Broker has the highest response rate (56%) and correspondingly the highest confirmed helpfulness (49%). But even in Broker, 35% of comments are unresolved. In Common and MSAL — where response rates are below 30% — nearly half of all comments are unresolved, meaning we cannot determine whether the AI's feedback was useful because engineers didn't evaluate it. +Every repository posts a precision above 96%. MSAL leads on both response rate (93%) and helpfulness (67%) with zero incorrect comments and only one unresolved. Broker carries the highest comment volume and the largest Declined bucket (16), reflecting a team that engages heavily and frequently makes deliberate design-tradeoff calls — those declines are not Copilot failures. Common sits in the middle. Notably, the "Not Helpful" column — the only one that reflects poorly on the AI — is at most a single comment in any repository. Coverage across the three repos: | Repository | Human PRs | PRs Reviewed by Copilot | Coverage | |------------|-----------|------------------------|----------| -| Common | 68 | 47 | 69% | -| MSAL | 31 | 19 | 61% | -| Broker | 64 | 47 | 73% | +| Common | 38 | 23 | 61% | +| MSAL | 14 | 7 | 50% | +| Broker | 35 | 23 | 66% | --- ## Results by Engineer -Each engineer has two GitHub accounts (a personal account for public repos and an EMU account for the private broker repo). These have been merged. Names are anonymized. +Each engineer may have two GitHub accounts (a personal account for public repos and an EMU account for the private Broker repo). These have been merged. Names are anonymized and ordered by helpfulness (descending), with comment volume as a tie-break. + +| Engineer | Comments | Replied | Response Rate | Helpful | Declined | Not Helpful | Unresolved | Helpfulness | Precision | +|----------|----------|---------|---------------|---------|----------|-------------|------------|-------------|-----------| +| **Engineer A** | 7 | 7 | **100%** | 7 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer B** | 3 | 3 | **100%** | 3 | 0 | 0 | 0 | **100%** | 100% | +| **Engineer C** | 11 | 9 | **81.8%** | 9 | 0 | 0 | 2 | **81.8%** | 100% | +| **Engineer D** | 6 | 5 | **83.3%** | 4 | 0 | 1 | 1 | **66.7%** | 80.0% | +| **Engineer E** | 30 | 23 | **76.7%** | 17 | 7 | 0 | 6 | **56.7%** | 100% | +| **Engineer F** | 9 | 5 | **55.6%** | 5 | 2 | 0 | 2 | **55.6%** | 100% | +| **Engineer G** | 11 | 6 | **54.5%** | 6 | 0 | 1 | 4 | **54.5%** | 85.7% | +| **Engineer H** | 6 | 3 | **50.0%** | 3 | 0 | 0 | 3 | **50.0%** | 100% | +| **Engineer I** | 4 | 4 | **100%** | 2 | 2 | 0 | 0 | **50.0%** | 100% | +| **Engineer J** | 19 | 15 | **78.9%** | 8 | 9 | 0 | 2 | **42.1%** | 100% | +| **Engineer K** | 19 | 15 | **78.9%** | 8 | 7 | 0 | 4 | **42.1%** | 100% | +| **Engineer L** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer M** | 3 | 0 | **0%** | 0 | 0 | 0 | 3 | **0%** | n/a | +| **Engineer N** | 1 | 0 | **0%** | 0 | 0 | 0 | 1 | **0%** | n/a | + +*Helpfulness = Confirmed Helpful / Total Comments. Precision = Helpful / (Helpful + Not Helpful), excluding Declined and Unresolved. Response Rate = Replied / Total Comments.* + +**Key observation — the Declined column changes how we read this table.** Under the old binary rubric, Engineer J (42.1% helpful, 9 declined) and Engineer K (42.1% helpful, 7 declined) would have looked like the engineers getting the *least* value from Copilot. But their large Declined buckets are not Copilot failures — they are experienced engineers who engage with nearly 80% of comments and make frequent, deliberate design-tradeoff calls. Both post a **100% precision**: Copilot was never actually wrong on their PRs. The real "not helpful" signal is tiny and concentrated — only Engineer D and Engineer G have a single incorrect comment each, and every other engineer who engaged has 100% precision. Engagement, not accuracy, remains the main lever: the engineers at a 0% response rate (Engineers L, M, and N) simply never evaluated the feedback, so it fell into the Unresolved bucket. -| Engineer | Comments | Replied | Ignored | Response Rate | Confirmed Helpful | Confirmed Not Helpful | Unresolved | Helpfulness | -|----------|----------|---------|---------|---------------|------|------|------|-------------| -| **Engineer A** | 20 | 20 | 0 | **100%** | 14 | 6 | 0 | **70.0%** | -| **Engineer B** | 83 | 75 | 8 | **90.4%** | 57 | 26 | 0 | **68.7%** | -| **Engineer C** | 15 | 4 | 11 | **26.7%** | 8 | 2 | 5 | **53.3%** | -| **Engineer D** | 40 | 30 | 10 | **75.0%** | 19 | 14 | 7 | **47.5%** | -| **Engineer E** | 110 | 37 | 73 | **33.6%** | 44 | 18 | 48 | **40.0%** | -| **Engineer F** | 99 | 20 | 79 | **20.2%** | 36 | 11 | 52 | **36.4%** | -| **Engineer G** | 63 | 22 | 41 | **34.9%** | 20 | 12 | 31 | **31.7%** | -| **Engineer H** | 100 | 24 | 76 | **24.0%** | 27 | 12 | 61 | **27.0%** | -| **Engineer I** | 24 | 6 | 18 | **25.0%** | 5 | 0 | 19 | **20.8%** | -| **Engineer J** | 3 | 1 | 2 | **33.3%** | 0 | 0 | 3 | **0%** | +--- -*Helpfulness = Confirmed Helpful / Total Comments. Response Rate = Replied / Total Comments. Unresolved = comments with no reply and no definitive diff evidence either way.* +## Response Behavior Deep Dive -**Key observation:** There is a strong correlation between response rate and helpfulness. Engineer A (100% response rate) and Engineer B (90%) have the highest helpfulness (70% and 69%) — and crucially, **zero unresolved comments**. When engineers engage, we know exactly what's helpful and what's not. Engineers with low response rates (Engineer F at 20%, Engineer H at 24%) have massive unresolved buckets (52 and 61 comments respectively) — over half their comments go into a black hole where we can't tell if the AI was right or wrong. +Of the 132 total comments: ---- +- **95 (72.0%) received a reply.** Of those, **69.5% were helpful**, **28.4% were declined** (correct but not adopted), and only **2.1% were incorrect**. When engineers engage, the overwhelming majority of Copilot feedback is either acted on or acknowledged as reasonable — and almost none of it is wrong. +- **37 (28.0%) were not replied to.** Of those, **16.2% were silently addressed** (verified via diff) and the remaining **83.8% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. -## Response Behavior Deep Dive +### What happens to comments engineers reply to -Of the 557 total comments: +| Reply verdict | Count | % of replied | Counts against Copilot? | +|---------------|-------|-------------|-------------------------| +| Helpful — accepted or fixed | 66 | 69.5% | No — positive | +| Declined — correct but not adopted | 27 | 28.4% | **No — neutral** | +| Not Helpful — incorrect | 2 | 2.1% | Yes | -- **239 (42.9%) received a reply.** Of those, **60.3% were helpful** and 39.7% were not helpful. When engineers engage, the majority of Copilot feedback turns out to be useful. -- **318 (57.1%) were ignored.** Of those, **27.0% were silently addressed** (verified via diff), **1.9% were on stale code** (confirmed not helpful), and the remaining **71.1% are unresolved** — we cannot determine whether the comment was useful because the engineer never evaluated it. +The takeaway is stark: among the 95 comments engineers actually engaged with, they judged Copilot *wrong* only twice. Everything else was either adopted or was a legitimate observation the engineer chose to handle differently. ### What happens to ignored comments | What happened | Count | % of ignored | Verdict | |---------------|-------|-------------|---------| -| Suggestion code silently applied (verified via diff) | 50 | 15.7% | Confirmed Helpful | -| Exact commented lines modified in subsequent commit | 7 | 2.2% | Confirmed Helpful | -| Re-audit: evidence of fix at nearby lines | 29 | 9.1% | Confirmed Helpful | -| Merged without any subsequent commits | 122 | 38.4% | **Unresolved** | -| File never modified after the comment | 9 | 2.8% | **Unresolved** | -| Comment on stale/outdated code | 6 | 1.9% | Confirmed Not Helpful | -| File modified but not at the commented lines | 45 | 14.2% | **Unresolved** | -| File modified but no line number to verify | 50 | 15.7% | **Unresolved** | +| Suggestion code / exact lines silently applied (verified via diff) | 6 | 16.2% | Confirmed Helpful | +| No reply and no diff evidence either way | 31 | 83.8% | **Unresolved** | -The single largest category is the **122 comments (38.4% of ignored)** where the engineer merged the PR without pushing any additional commits after Copilot's review. These represent the final review round being skipped entirely — the feedback had zero chance of impact regardless of its quality. +The 31 unresolved comments are the only real blind spot in this report. They are not confirmed failures — they are feedback that entered a void. If engineers engaged with even half of them, precision and helpfulness would both firm up further. --- ## What Copilot Is Good At -The most common categories of helpful comments, with real examples from our PRs: +The most valuable helpful comments this period, with real examples from our PRs: -**Catching real bugs:** -> *PR #3050 (Common):* Copilot flagged that `"$it"` string wrapping doesn't JSON-escape the content, which could break consumers if contract values contain special characters. -> *Engineer reply: "You're right. Making the change."* +**Catching real initialization-ordering bugs:** +> *PR #215 (Broker) — `AriaInitializer`:* Copilot flagged that a flight/telemetry provider was being read before it was initialized, an ordering problem that would silently drop data. +> *Engineer reply: "Good catch — this is a fundamental ordering problem. Fixing it."* +> +> This is exactly the kind of subtle, cross-method sequencing issue that is easy to miss in review but expensive in production. Copilot caught it from the diff alone. -**Stale documentation and naming inconsistencies:** -> *PR #64 (Broker):* Copilot identified four locations where KDoc still referenced the old flight constant `USE_TEE_ONLY_FOR_TOKEN_BINDING` after it was renamed to `USE_TEE_ONLY_FOR_HARDWARE_BOUND_KEYS`. All four were silently fixed in the next commit. +**Telemetry schema and contract discrepancies:** +> *PR #204 (Broker) — `AccountChooser`:* Copilot noticed the emitted telemetry attribute didn't match the schema described in the PR, flagging a contract mismatch. +> *Engineer acknowledged and reconciled the schema.* -**Dead code and unused imports:** -> *PR #3040 (Common):* Copilot spotted an unused local variable `enabledSettingRaw` that was assigned but never read. Verified as fixed via diff analysis — the suggested replacement code appeared in the commit additions. +**Deprecated-API and compatibility issues:** +> *PR #3147 (Common):* Copilot pointed out a newly-added overload shadowed a deprecated one and recommended `@JvmOverloads` / a default argument to preserve binary compatibility for Java callers. +> *Engineer reply: "Fixed. Thanks for pointing this out."* -**CI/pipeline configuration issues:** -> *PR #3038 (Common):* Copilot warned that using `vmImage: 'windows-latest'` makes the CD pipeline non-deterministic. The engineer changed to a pinned image version. +**Security and policy consistency:** +> *PR #3156 (Common) — OWASP dependency check:* Copilot identified an inconsistency in how a CVE suppression was applied across modules. +> *Engineer reply: "Implemented in commit a58b52dc9; remaining remediation tracked in AB#3667951."* +> +> Copilot's comment didn't just get a thumbs-up — it produced a concrete fix and a tracked follow-up work item. -**The `@copilot apply` workflow:** -> In 16 instances (2.9%), engineers validated Copilot's feedback and then delegated the fix back to Copilot with: `@copilot open a new pull request to apply changes based on [this feedback]`. This is an efficient pattern where AI identifies and fixes the issue end-to-end. +**Behavior-vs-description correctness:** +> *PR #3165 (Common) — Authority validation:* Copilot noted a URL comparison was matching on the full URL when only the host should matter. +> *Engineer reply: "Good point — updated the code to check host only."* --- -## What Copilot Struggles With +## When Copilot Was Actually Wrong -The most common categories of unhelpful comments, with real examples: +Only **two** comments this period were genuine errors. We reproduce both in full, because at 1.5% they are the exception that proves the rule: -**Lacking domain context:** -> *Copilot:* "`shared_device_id` could be used for tracking across apps — consider hashing before emission." -> *Engineer:* "The shared_device_id is a random UUID generated by one of the participating apps and is not PII." +**Misjudging a library's behavior:** +> *PR #3171 (Common):* Copilot warned that a Mockito varargs matcher would not behave as intended and could cause the test to pass incorrectly. +> *Engineer reply: "Verified this isn't reproducible on Mockito 5.11.0 — the suite passes 12/12."* > -> Copilot applied a general security heuristic without understanding that this particular identifier is already random and non-linkable. +> Copilot applied a heuristic that held for older Mockito versions but not the one actually in use. A version-specific false positive. -**Suggesting tests for trivial code:** -> *Copilot:* "New telemetry attributes lack test coverage..." -> *Engineer:* "These are just telemetry related changes and adding unit tests will be overdo here." +**Hallucinating a missing annotation:** +> *PR #212 (Broker):* Copilot claimed a method was missing an `@Override` annotation. +> *Engineer reply: "The override annotation is already there."* > -> This was a recurring theme — Copilot frequently requests tests for logging/telemetry code that the team considers low-risk. +> The annotation was present in the code. This is a straightforward hallucination — the only category of error we truly want to drive to zero. -**Misunderstanding APIs:** -> *Copilot:* "`00000003-0000-0ff1-ce00-000000000000` is the resource ID for Microsoft Graph, not SharePoint Online." -> *Engineer:* "00000003-0000-0ff1-ce00-000000000000 is SharePoint Online." +--- + +## When Copilot Was Right but the Engineer Declined + +These comments were **not wrong** — the engineer understood the point and deliberately chose a different path. Under the old methodology, all of these would have been miscounted as "not helpful." They are the heart of this report's fairness correction. + +**Deliberate notation choice (the flagship case):** +> *PR #201 (Broker):* Copilot flagged the code comment `~1 per device` as potentially misleading, since the tilde could be read as an exact value. +> *Engineer reply: "~ = approximate. i think this is ok."* +> +> A perfectly reasonable readability observation. The author simply judged the notation clear enough in context. Copilot had no way to know that tolerance beforehand — this is neutral, not a failure. + +**Subjective naming preference:** +> *PR #3151 (Common):* Copilot suggested encoding the "prompt" semantics into a span name for clarity. +> *Engineer reply: "I think the current name is fine."* > -> Copilot was factually wrong about a well-known Microsoft service resource ID. +> A judgment call on naming. Both names are defensible; the author preferred the existing one. -**Commenting on intentional design choices:** -> *Copilot:* "`getPackageInfo() != null` is redundant since it either returns PackageInfo or throws..." -> *Engineer:* "This is fine. The verbosity makes the code clearer to understand." +**Intentional verbosity for a new feature:** +> *PR #223 (Broker):* Copilot suggested not logging a full stack trace for an expected exception type. +> *Engineer reply: "Since this is a new feature, we want that stack trace for all exceptions."* +> +> The engineer deliberately wants maximum diagnostic detail while the feature is new. Copilot's general "don't log noisy stack traces" heuristic was sound but didn't apply here. -**Over-engineering suggestions:** -> *Copilot:* "Use a bounded min-heap of size `PRT_ARTIFACT_LIMIT` to reduce overhead..." -> *Engineer:* (The list can only ever contain 3 items — there are only 3 broker apps on Android.) +**Context Copilot couldn't have known:** +> *PR #3177 (Common):* Copilot suggested removing an INFO-level log as noise. +> *Engineer reply: "We need this to catch a regression in the COBO/COPE scenarios."* +> +> The log is intentionally there to detect a specific device-management regression — context that isn't visible in the diff. --- -## Most Reviewed Files +## Trend Over Time + +This is the fourth analysis period. Response rate has recovered and comment volume has stabilized. Because the **Declined vs. Incorrect** split and the **Precision** metric were introduced this period, prior periods show only the old combined "Dismissed" figure — so the current period is broken out below the table. + +| Period | Days | Comments | Comments/wk | Response Rate | Helpful | Dismissed (combined) | Unresolved | +|--------|------|----------|-------------|---------------|---------|----------------------|------------| +| Jan 24 – Mar 25 | 60 | 570 | 66.3 | 44.4% | 38.6% | 17.4% | 44.0% | +| Mar 24 – Apr 19 | 26 | 170 | 45.9 | 78.8% | 62.9% | 21.2% | 15.9% | +| Apr 20 – Jun 9 | 50 | 157 | 22.1 | 67.5% | 52.9% | 16.6% | 30.6% | +| **Jun 10 – Jul 15** | **35** | **132** | **26.4** | **72.0%** | **51.5%** | **22.0%** | **26.5%** | -| Rank | File | Comments | -|------|------|----------| -| 1 | `.github/workflows/copilot-issue-response.yml` | 27 | -| 2 | `broker4j/.../AttributeName.java` | 19 | -| 3 | `.github/workflows/copilot-ci-feedback.md` | 16 | -| 4 | `common/.../AuthorizationFragment.java` | 15 | -| 5 | `broker4j/.../MultipleWorkplaceJoinDataStore.java` | 11 | -| 6 | `broker4j/.../AbstractBrokerController.java` | 11 | -| 7 | `common/.../AzureActiveDirectoryWebViewClient.java` | 11 | -| 8 | `AADAuthenticator/.../BrowserSsoProvider.kt` | 10 | -| 9 | `broker4j/.../BrokerFlight.java` | 9 | -| 10 | `broker4j/.../DeviceRegistrationRequestHandler.java` | 9 | +**What changed this period:** -CI/workflow files and large Java classes with many change touchpoints attract the highest volume of comments. `AttributeName.java` appears frequently because many PRs add telemetry attributes. +- **Response rate rose from 67.5% to 72.0%**, driven partly by engineers going back to reply to previously-unanswered comments. +- **Trend-series helpfulness dipped slightly** (52.9% → 51.5%) while unresolved comments dropped (30.6% → 26.5%) — more feedback is being evaluated, even though normalized helpfulness was broadly stable. +- **The "Dismissed" number is misleading in isolation.** This period's combined dismissal rate (22.0%) looks comparable to prior periods, but the new split reveals that **20.5 of those 22.0 points are Declined (Copilot was right)** and only **1.5 points are genuine errors**. Copilot's **precision this period is 97.3%**. Earlier periods almost certainly had a similar breakdown — we simply couldn't see it under the binary rubric, which means historical "not helpful" rates of 16–21% substantially overstated how often the AI was actually wrong. + +*Note: the Declined/Incorrect split and Precision metric are only available from this period onward. For apples-to-apples trend comparison, the table shows the combined dismissal rate; the current period's true error rate is 1.5%.* + +*Trend continuity note: the historical series uses the same automated diff-evidence rules across all four periods (68 helpful / 35 unresolved this period). The authoritative report body additionally applies four manually re-audited diff flips, producing 72 helpful / 31 unresolved. This deliberate distinction preserves an apples-to-apples trend while keeping the current body classification maximally accurate.* --- ## Key Takeaways -1. **57% of Copilot review comments receive no response from engineers.** This is the most significant finding. The majority of AI review feedback is never acknowledged. Some of it is silently acted on (27% of ignored comments show diff evidence of a fix), but the vast majority — 71% of ignored comments — are unresolved. We simply don't know if they were useful because no one evaluated them. +1. **Copilot is almost never wrong — precision is 97.3%.** Of 132 comments, only 2 were genuine errors. When engineers engaged deeply enough for us to judge correctness, Copilot was right 72 times out of 74. + +2. **"Declined" is not "wrong," and it's now counted fairly.** 27 comments (20.5%) were correct feedback the engineer deliberately chose not to adopt. Under the old binary rubric these would have dragged down Copilot's apparent quality. They are neutral — the AI had no way to know the author's intent. + +3. **Response rate climbed to 72%.** Up from 68% last period and 44% in the first analysis. Several engineers returned to reply to comments they had previously ignored — one reaching a 100% response rate across all seven of their comments. + +4. **54.5% of comments led to a confirmed improvement.** With only 1.5% confirmed wrong, the helpfulness ceiling is high; the remaining gap is the 23.5% unresolved comments that no one evaluated. -2. **41% of all comments led to a confirmed code improvement.** But 41% are unresolved, meaning the true helpfulness rate lies between 41% (floor) and 82% (ceiling). We cannot narrow this range without engineer engagement. +5. **Every repository posts 96%+ precision.** MSAL 100%, Broker 97.1%, Common 96.7%. The "not helpful" column never exceeds one comment per repo. -3. **When engineers engage, 60% of comments are helpful.** Of the 239 comments that received a reply, 144 (60%) led to acknowledged improvements. This suggests the AI review quality itself is decent — the bottleneck is adoption, not accuracy. +6. **The Declined bucket concentrates in high-engagement engineers.** Engineer J (9 declined) and Engineer K (7 declined) engage with ~80% of comments and make frequent design-tradeoff calls — both at 100% precision. Their lower "helpfulness" percentage is not a Copilot quality problem. -4. **Only 18% of comments are confirmed not helpful.** When we restrict "not helpful" to comments where the engineer explicitly dismissed the feedback — the only cases where we have positive evidence of low quality — the rate is surprisingly low. +7. **Unresolved is the only real blind spot.** 31 comments (23.5%) were never evaluated. This is the one lever entirely in the team's hands: engaging with even half of them would tighten every metric in this report. -5. **Engagement is the strongest predictor of value.** Engineers who reply to 75%+ of comments see 47-70% confirmed helpfulness with zero unresolved comments. Engineers who reply to <35% see 20-40% helpfulness with massive unresolved buckets (50-60% of their comments). The tool works better when engineers work with it. +8. **The bottleneck is adoption and engagement, not accuracy.** The data is now unambiguous: Copilot's review quality is high. The variance in per-engineer "value" is almost entirely explained by how often each engineer reads and responds to the feedback. -6. **38% of ignored comments are on the final commit before merge.** These 122 comments represent the last review round being skipped entirely — the engineer merged without pushing any further changes. These comments may have been perfectly valid, but the feedback had zero chance of impact. +--- + +## What We'd Recommend + +Based on this analysis, we see three opportunities: -7. **Broker gets the most value (49%), Common is middling (36%), MSAL is lowest (26%).** This correlates with response rate: Broker engineers reply to 56% of comments, while Common (30%) and MSAL (25%) reply far less. +**1. Keep engaging — it's working.** Response rate rose to 72% this period, and the payoff is visible: we can now tell that Copilot is wrong only 1.5% of the time. The engineers who went back to reply to old comments materially improved the fidelity of this report. A quick reply — even a decline with a one-line reason — is what lets us separate "Copilot was wrong" from "I chose otherwise." That distinction is the entire value of this measurement. -8. **At ~5 comments per PR, the signal-to-noise conversation can't be settled without engagement.** We confirmed ~2 useful comments per PR on average, but with 41% of comments unresolved, the actual number could be higher. The only way to know is for engineers to evaluate the feedback. +**2. Close the unresolved gap before merge.** The 31 unresolved comments (23.5%) are feedback no one evaluated. Building a habit of scanning Copilot's last review round before clicking merge would convert most of these into a clear helpful/declined/incorrect verdict and give the feedback a chance to have impact. + +**3. Target the two real error patterns.** The only genuine errors this period were a version-specific library assumption (Mockito) and a hallucinated annotation. Both are addressable by refining `copilot-instructions.md` — e.g., noting the Mockito version in use and reminding the reviewer to confirm an annotation is actually absent before flagging it. Because genuine errors are already so rare, small instruction tweaks can plausibly drive them to zero. --- ## Methodology Notes -- **Scope.** Only inline code review comments from the `Copilot` and `copilot-pull-request-reviewer[bot]` users were counted. PR-level summary comments (205 total) were excluded from the helpfulness analysis. -- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent) were excluded. Only PRs authored by human engineers were analyzed. -- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` (addition) lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped with the comment's target line range (±5 line tolerance). This is a conservative approach — some fixes that refactored code differently than suggested may be missed. -- **AI-assisted reply classification.** Every comment that could not be definitively classified by automated methods was individually read and classified by the AI conducting this analysis (Claude), based on the reply text and domain context. These AI classifications were reviewed for accuracy by the report author but were not independently verified by the original PR engineers. -- **Conservative approach.** We only classify a comment as "Confirmed Helpful" when there is positive evidence: an explicit engineer acknowledgment, or verified code changes at the exact lines/tokens suggested. We only classify as "Confirmed Not Helpful" when there is positive evidence that the comment was *wrong or irrelevant*: an explicit engineer dismissal with a stated reason. Comments where the engineer simply did not engage — including the final review round that was merged without evaluation, files that were never modified, and files modified at different lines — are classified as "Unresolved" rather than assumed to be unhelpful. -- **Account merging.** Engineers with separate public GitHub accounts and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. -- **Data availability.** Raw data for all 557 comments (including comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. +- **Scope.** Only inline code review comments from the `Copilot` user were counted. PR-level summary comments were excluded from the helpfulness analysis. +- **Bot exclusions.** PRs authored by `copilot-swe-agent` (Copilot coding agent), `dependabot[bot]`, and `github-actions[bot]` were excluded. Only PRs authored by human engineers were analyzed. +- **Three-way reply classification.** Every replied comment was individually read and classified as Helpful, Declined, or Not Helpful (incorrect) by the AI conducting this analysis, based on the reply text and domain context. The dividing line between Declined and Not Helpful is **whether Copilot was factually/technically wrong**, not whether the engineer acted. These classifications were reviewed for accuracy by the report author but were not independently re-verified by the original PR engineers. +- **Precision.** Defined as Helpful ÷ (Helpful + Not Helpful-incorrect). Declined and Unresolved comments are excluded from both numerator and denominator because neither represents a Copilot mistake. +- **Diff verification.** For suggestion blocks, we extracted the suggested code tokens and checked if they appeared as `+` lines in the compare diff between the comment's commit and the PR head. For prose comments, we checked if the diff hunk line ranges overlapped the comment's target line range (±5 line tolerance). This is conservative — some fixes that refactored code differently than suggested may be missed. +- **Conservative "Unresolved."** Comments with no reply and no diff evidence are classified as Unresolved rather than assumed unhelpful. "Unresolved" ≠ "Not Helpful." +- **Account merging.** Engineers with separate public GitHub and EMU (Enterprise Managed User) accounts were merged based on known identity mappings. +- **Data availability.** Raw data for all 132 comments (comment text, engineer replies, diff verification evidence, and final verdicts) is stored at `%TEMP%\copilot-review-analysis\` for independent verification. --- -*Analysis conducted March 23-24, 2026. Data covers all PRs created January 23 – March 23, 2026 in the Common, MSAL, and Broker repositories.* +*Analysis conducted July 15, 2026. Data covers all PRs created June 10 – July 15, 2026 in the Common, MSAL, and Broker repositories. This report introduces a three-way verdict methodology (Helpful / Declined / Not Helpful) and a Copilot precision metric to fairly distinguish AI errors from deliberate engineer judgment calls.* + diff --git a/.github/skills/copilot-review-analyst/references/classification-rules.md b/.github/skills/copilot-review-analyst/references/classification-rules.md index 7685b431..644af53e 100644 --- a/.github/skills/copilot-review-analyst/references/classification-rules.md +++ b/.github/skills/copilot-review-analyst/references/classification-rules.md @@ -4,10 +4,25 @@ Guide for the AI agent performing Phase 3 reply classification. Read each replie ## Phase 3: Classifying Replied Comments -For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of: +For every comment where `HasReply = true`, read the `CommentBody` (what Copilot said) and `HumanReplyText` (what the engineer replied), then assign one of **three** verdicts: - **`helpful`** — The engineer's reply indicates Copilot's feedback led to (or will lead to) a code improvement -- **`not-helpful`** — The engineer's reply indicates Copilot's feedback was wrong, irrelevant, or not actionable +- **`not-helpful`** — Copilot's feedback was **factually wrong** (a false positive, hallucination, or a demonstrable error about the code/library/framework). This is the only bucket that counts *against* Copilot's review quality. +- **`declined`** — Copilot's feedback was **correct or reasonable**, but the engineer **intentionally chose not to act** on it for a by-design, subjective, or contextual reason (or the code became moot). This is **neutral** — it is *not* counted against Copilot, because Copilot had no way to know the author's intent beforehand. + +### Why `declined` exists (the fairness principle) + +A dismissal is **not** evidence that Copilot was wrong. Copilot reviews a diff without the author's design intent, offline discussions, or downstream context. When an engineer replies "intentional" or "this is fine as-is," Copilot's observation was often perfectly valid — the engineer simply made a judgment call. Counting these against Copilot punishes it for not being a mind-reader and understates real review quality. + +So we split the old "not-helpful" bucket into two: +- **`not-helpful` (incorrect)** — Copilot was actually *wrong*. Fair to count against it. +- **`declined` (by-design / subjective / moot)** — Copilot was *right or reasonable*, engineer declined. Neutral. + +The **Copilot precision** metric measures correctness only where it is evaluable: + +> **precision = helpful ÷ (helpful + not-helpful-incorrect)** + +`declined` and `unresolved` are excluded from both numerator and denominator. Precision answers: *"When Copilot spoke and we could judge correctness, how often was it actually right?"* ### What counts as Helpful @@ -18,26 +33,88 @@ For every comment where `HasReply = true`, read the `CommentBody` (what Copilot - **Linked a commit**: Reply contains a commit SHA or link showing they applied a fix - **Acknowledged for future**: "this can be considered in another PR" (acknowledges the issue is valid) -### What counts as Not Helpful +### What counts as Not Helpful (Copilot incorrect) + +Reserve this bucket for cases where the engineer's reply demonstrates Copilot was **factually wrong**: -- **Explicit dismissal**: "won't fix", "by design", "intentional", "not applicable", "false positive", "not relevant" -- **Copilot was wrong**: "incorrect", "Copilot is wrong", "hallucinating", "not accurate", "misunderstanding" -- **Already handled**: "already done", "already handled", "this is fine" -- **Explained away**: Engineer explains why the suggestion doesn't apply — "this is just telemetry", "we consciously chose this", "legacy code", "overdo", "only used in X context", "can't happen" -- **Dismissed or outdated**: "outdated", "dismissed", "out of scope" +- **Disproven claim**: Engineer verified the opposite — "Verified this isn't reproducible on Mockito 5.11.0; the suite passes 12/12" (Copilot was wrong about library behavior) +- **Hallucination**: Copilot flagged something that isn't there — "The `@Override` annotation is already there" (Copilot claimed it was missing) +- **Wrong about the code/framework/version**: Copilot's premise is objectively false for this codebase +- **False positive with proof**: Engineer shows the flagged issue cannot occur + +If the engineer *demonstrates* Copilot got a fact wrong → `not-helpful`. + +### What counts as Declined (neutral — Copilot correct/reasonable, engineer declined) + +- **By-design**: "intentional", "by design", "this is intentional", "we consciously chose this", "keeping as-is" +- **Subjective / nitpick**: "too nitpicky", "the current name is fine", "minor, no-fix", "imo this reads fine" +- **Contextual reason to keep**: "we need this log to catch a regression in cobo/cope", "for a new feature we want the full stack trace" +- **Acceptable risk**: "this should be okay since it's in a log file" +- **Moot / obviated**: "not relevant now that we removed this change", "outdated" (the code changed so the comment no longer applies — Copilot wasn't *wrong*, just stale) +- **Deprecated target**: "this config is deprecated, not fixing" +- **Accurate but declined**: engineer explicitly says the comment *was* accurate but chose not to act ("this is outdated, but was accurate") + +The test: *Did the engineer demonstrate Copilot was wrong (→ not-helpful), or did they acknowledge/ignore a valid-but-unwanted point (→ declined)?* When in doubt between `not-helpful` and `declined`, prefer **`declined`** unless there is positive evidence Copilot was factually incorrect. ### Edge Cases - **Mixed signals** (both positive and negative in same reply): Read the full reply to determine the engineer's overall intent. Don't rely on individual words — understand the sentence. -- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **not-helpful** if they're brushing it off. -- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer whether the engineer is acknowledging or dismissing. When genuinely unclear, lean toward **not-helpful** (conservative). +- **Administrative replies** ("will consider later", "not for this PR"): Classify as **helpful** if they acknowledge the issue is valid but defer it; classify as **declined** if they're setting it aside as by-design/out-of-scope (not because Copilot was wrong). +- **Short/ambiguous replies** ("ok", "noted", "see above"): Use the Copilot comment context to infer intent. If it reads as acknowledgment → helpful. If it reads as a soft dismissal of a valid point → declined. Only use **not-helpful** when there is evidence Copilot was factually wrong. +- **"Outdated"**: Treat as **declined** (the code moved on; not a Copilot correctness failure) unless the engineer indicates Copilot's underlying claim was itself wrong. **Before scoring, apply the Force-Push Confound check below** — an "outdated" marker on a rewritten snapshot can hide a comment that was *correct* when Copilot reviewed it, and must never be recorded as `not-helpful`. ### Important: Read the Full Reply Do NOT use simple keyword matching. Read the engineer's full reply in context. For example: - "This won't fix the actual issue we're seeing" — This is NOT a "won't fix" dismissal; the engineer is discussing a different topic -- "Thanks but this is intentional" — Despite "thanks", this is a dismissal +- "Thanks but this is intentional" — Despite "thanks", this is a **declined** (by-design), not a Copilot error - "I disagree with this specific suggestion but good catch on the typo above" — Mixed; classify based on the primary concern +- "~ = approximate, I think this is ok" — Copilot's readability nit was reasonable; engineer made a judgment call → **declined**, not not-helpful + +## The Force-Push Confound (mandatory validation before scoring) + +**A "correct" Copilot comment can look like a false positive when the engineer rewrites history.** If an engineer rebases or force-pushes *after* Copilot reviews — a common habit right after opening a PR — the commit Copilot reviewed is orphaned. GitHub then marks the comment **"outdated"** and the visible code no longer matches what Copilot saw, so a comment that was *accurate at the reviewed snapshot* reads as stale or even hallucinated. Left unchecked, this silently inflates the `declined`/`unresolved` buckets and, in the worst case, scores a **correct** comment as `not-helpful` — understating precision. + +> This was not hypothetical: it caused the single decisive misclassification in the Jul 2026 audit (broker #212's `@Override` comment was mislabeled a hallucination; the `@Override` was genuinely absent at the reviewed commit `bcf360ea`, which the engineer force-pushed away ~2 min later). Precision moved **up** once corrected. + +### Detection — three immutable fields + +Phase 1 (`analyze.ps1`) now records these per comment in `raw_results.json`: + +- **`OriginalCommitId`** — `original_commit_id`, the exact commit Copilot reviewed. Immutable; survives force-push. +- **`ReviewedCommitRewritten`** — `true` when `OriginalCommitId` is **absent** from the PR's current commit list (`/pulls/{n}/commits`), i.e. the reviewed snapshot was rewritten. This is the force-push fingerprint. +- **`DiffHunk`** — `diff_hunk`, a frozen copy of the exact lines Copilot saw. Ground truth for what Copilot was actually looking at. + +`precise.ps1` carries `ReviewedCommitRewritten` forward and flags any `not-applied`/`file-not-changed` verdict that sits on a rewritten snapshot. + +### The rule + +For **every** comment classified `declined`, `not-helpful`, `unresolved`, or a no-reply `not-applied`/`file-not-changed` verdict, **first check `ReviewedCommitRewritten`**: + +1. If `false` → classify normally (no confound). +2. If `true` → the dismissal is **unsafe to credit as-is**. Re-derive Copilot's claim from the immutable `DiffHunk` (and, if needed, the reviewed file via `contents?ref=`) and compare it to the **merged** state: + - Copilot's claim was **true at the reviewed snapshot** and the merged code already satisfies it → the fix was effectively adopted by the rewrite → **helpful** (never `not-helpful`, never `declined`). + - Copilot's claim was **true at the reviewed snapshot** and the engineer explicitly declined it on the merits ("intentional", "okay in a log file") → **declined** (the rewrite was incidental). + - Copilot's claim was **false even at the reviewed snapshot** → `not-helpful` (a genuine error). + +**Never assign `not-helpful` to a comment on a rewritten snapshot without confirming Copilot was wrong at `OriginalCommitId`.** A stale marker is not evidence of a Copilot error. + +### Precision is computed only after this validation + +Because a mis-scored force-push case lands directly in the precision denominator, **precision must be recomputed only after the force-push validation pass has run** over all `declined`/`not-helpful`/`unresolved` comments. Report the count of `ReviewedCommitRewritten = true` comments as a measurement-integrity note. + +### Reproducible check (per comment) + +```bash +# 1. The commit Copilot reviewed (immutable): +gh api repos//pulls/comments/ --jq '.original_commit_id' +# 2. Is it still in the PR's commit list? (empty output => rewritten away => confound) +gh api repos//pulls//commits --paginate --jq '.[].sha' | grep +# 3. Ground truth Copilot saw: +gh api repos//pulls/comments/ --jq '.diff_hunk' +# 4. (optional) The reviewed file as Copilot saw it: +gh api "repos//contents/?ref=" --jq '.content' | base64 -d +``` ## Phase 3 Output Format @@ -51,11 +128,11 @@ Map of comment ID to verdict for every replied comment: { "1234567890": "helpful", "1234567891": "not-helpful", - "1234567892": "helpful" + "1234567892": "declined" } ``` -Keys are comment IDs (as strings). Values are `"helpful"` or `"not-helpful"`. +Keys are comment IDs (as strings). Values are `"helpful"`, `"not-helpful"` (Copilot incorrect), or `"declined"` (Copilot correct/reasonable but engineer declined). ### `reaudit-flips.json` @@ -81,7 +158,9 @@ These are handled by `precise.ps1` and `final-classification.ps1` automatically: | `suggestion-applied`, `suggestion-likely-applied`, `exact-lines-modified` | **Helpful** | | `lines-modified-different-fix` | **Helpful** | | `file-changed-elsewhere`, `file-changed-no-line-info` | **Not Helpful** (unless in re-audit flips) | -| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** | +| `file-not-changed`, `no-subsequent-commits`, `not-applied` | **Not Helpful** (unless in re-audit flips) | + +> **Force-push caveat:** any `not-applied` / `file-not-changed` row with `ReviewedCommitRewritten = true` is UNSAFE to score as Not Helpful — the reviewed snapshot was rewritten, so the fix may live in the orphaned commit. Validate against the immutable `DiffHunk` (see "The Force-Push Confound") before finalizing; flip to **Helpful** if Copilot's claim was already satisfied by the merged code. ## Account Mapping diff --git a/.github/skills/copilot-review-analyst/references/report-formatting.md b/.github/skills/copilot-review-analyst/references/report-formatting.md index b74db5b4..00961c2f 100644 --- a/.github/skills/copilot-review-analyst/references/report-formatting.md +++ b/.github/skills/copilot-review-analyst/references/report-formatting.md @@ -11,18 +11,20 @@ Generate both formats. Templates are in `assets/` within this skill folder. | # | Section | Content | Depth | |---|---------|---------|-------| | 1 | **Background** | Team context, what repos are covered, what was enabled | 2-3 prose paragraphs | -| 2 | **At a Glance** | 4 summary cards (no-response %, helpful %, not-helpful %, unresolved %) + callout about adoption | Cards + 1 detailed callout box | -| 3 | **Overall Results** | Response rate bar, helpfulness verdict bar, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + 2 breakdown tables | -| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/not/unresolved) | Bar per repo + data table + 1 interpretive paragraph | +| 2 | **At a Glance** | 5 summary cards (helpful %, declined %, incorrect %, unresolved %, precision) + callout explaining the fairness correction | Cards + 1 detailed callout box | +| 3 | **Overall Results** | Response rate bar, four-way verdict bar, precision formula, breakdown tables | Narrative paragraph before each visual + verdict definitions table + yellow warning callout + breakdown tables | +| 4 | **Results by Repository** | Per-repo bars + table (comments, response rate, helpful/declined/incorrect/unresolved/precision) | Bar per repo + data table + 1 interpretive paragraph | | 5 | **Results by Engineer** | Table with colored columns (anonymize names for org-wide sharing) | Full table + blue callout box highlighting the engagement-value correlation | | 6 | **Response Behavior Deep Dive** | What happens to ignored comments (silently applied, merged without commits, etc.) | Summary stats + detailed breakdown table + interpretive paragraph | | 7 | **What Copilot Is Good At** | 4-5 real examples with PR references and engineer quotes | Each example: category header + full Copilot comment text (not truncated) + engineer reply + 1-2 sentence explanation | -| 8 | **What Copilot Struggles With** | 4-5 real examples showing false positives, domain gaps | Same format as above — full quotes + explanatory context | +| 8 | **When Copilot Was Wrong / When Feedback Was Declined** | Genuine errors and correct-but-declined examples in separate subsections | Full quotes + explanatory context; never present a deliberate decline as a Copilot error | | 9 | **Most Reviewed Files** | Top 10 files by comment count | Table + 1 interpretive paragraph | | 10 | **Key Takeaways** | 7-8 numbered findings | Each finding: bold stat + explanatory sentence | | 11 | **Recommendations** | 3 actionable next steps | Each recommendation: 1 full prose paragraph (not a bullet point) with reasoning | | 12 | **Methodology Notes** | How data was collected, classified, and validated | 5-6 bullet points with sufficient detail for reproducibility | +**Headline framing (all formats).** The hero band and the first Key Takeaway lead with **Helpful & adopted %** (adoption), *not* precision. Adoption answers "did engineers act on the feedback?" — the question leadership cares about — and precision is presented as the supporting second number. Leading with precision (which is near-100% by construction once Declined/Unresolved are excluded) overstates the story and buries the adoption signal. + ## Trend Section (Section 2.5 — between At a Glance and Overall Results) **Only generated when `history.json` has ≥2 entries.** Skip entirely on the first run. @@ -35,11 +37,12 @@ Load `~/.copilot-review-analysis/history.json`. Entries are sorted newest-first. Since periods may have different lengths (e.g., 60 days vs 14 days): -1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, not-helpful %, unresolved %, and replied-helpful rate are directly comparable across any period length. +1. **Compare rates/percentages, not absolute counts.** Response rate, helpful %, combined dismissal %, unresolved %, and replied-helpful rate are directly comparable across any period length. For snapshots created after the three-way methodology change, also show Declined %, Incorrect %, and Precision. 2. **Show counts as context only.** Display total comments alongside `comments/week` for normalized volume comparison. Do NOT compute count deltas like "comments dropped from 570 to 85" — this is misleading when periods differ. 3. **Show period duration prominently.** Every trend row must include the date range and duration (e.g., "Jan 24–Mar 25 (60d)"). 4. **Use "pp" (percentage points) for deltas.** "↑ +7.6pp" not "↑ +7.6%". The delta is the arithmetic difference between two percentages. 5. **Color-code deltas.** Green (↑) for improvements (response rate up, helpful up, not-helpful down, unresolved down). Red (↓) for regressions. +6. **Preserve trend continuity.** Historical snapshots created before the `declined` verdict only have a combined dismissal rate. Label that column **Dismissed (combined)** and explain that the current period can be split into Declined (neutral) and Incorrect (counts against Copilot). Do not retroactively infer the split for old periods. ### Markdown Format (2 runs — current vs previous) @@ -98,10 +101,19 @@ From `final_classification.json`: # Overall $total = $data.Count $helpful = ($data | Where-Object { $_.Verdict -eq "helpful" }).Count -$notHelpful = ($data | Where-Object { $_.Verdict -eq "not-helpful" }).Count -$unresolved = $total - $helpful - $notHelpful +$declined = ($data | Where-Object { $_.Verdict -eq "declined" }).Count +$incorrect = ($data | Where-Object { + $_.Replied -eq $true -and $_.Verdict -eq "not-helpful" +}).Count +$unresolved = ($data | Where-Object { + $_.Replied -eq $false -and $_.Verdict -eq "not-helpful" +}).Count $replied = ($data | Where-Object { $_.Replied -eq $true }).Count $responseRate = [math]::Round(($replied / $total) * 100, 1) +$precisionDenominator = $helpful + $incorrect +$precision = if ($precisionDenominator -gt 0) { + [math]::Round(($helpful / $precisionDenominator) * 100, 1) +} else { 0 } # Per-repo $repoStats = $data | Group-Object Repo | ForEach-Object { ... } @@ -112,10 +124,12 @@ $engStats = $data | Group-Object Engineer | ForEach-Object { ... } ## Outlook HTML Formatting Rules +> **Canonical design:** `assets/Copilot-Code-Review-Effectiveness-Report-Outlook.html` is the current **v3 newsletter** template — read it in full before generating. The primitives below (table layout, inline styles, `bgcolor`+inline pairing, bars-not-SVG) still hold; the load-bearing v3 techniques (single-row chips, edge-to-edge frame, newsletter masthead, `mso-line-height-rule`) are documented under **v3 Newsletter Rendering** below. The asset is written entirely single-quoted so it stays escaping-free (`"`/`\` count = 0) for a trivial Graph push. + Outlook strips most modern CSS. Follow these rules strictly: ### Layout -- Wrap entire body in a centered `` for consistent margins +- The current shell is a **1200px hybrid-fluid** newsletter (VML hero fallback + stacking media query), not a fixed centered table. Older reports used a centered `
`; the v3 shell supersedes it. Do not hand-roll the shell — reuse the asset's outer structure. - Use **table-based layouts only** — no flexbox, no grid, no float - All styles must be **inline** — Outlook strips `