Report MSTest `[Retry]` attempts to the platform instead of discarding them by Evangelink · Pull Request #10295 · microsoft/testfx

Evangelink · 2026-07-28T18:11:42Z

Draft implementation of #10292.

Problem

UnitTestRunner.RunSingleTestAsync kept only the last retry attempt:

result = retryResult.TryGetLast() ?? throw ApplicationStateGuard.Unreachable();

so an in-process [Retry] was invisible everywhere — the terminal showed a plain passing test, and the CTRF report claimed flaky: 0 with no retryAttempts[] even though CtrfReportEngine.CollapseAttempts / IsFlaky already implement the correct rule.

Approach

Publish every attempt as its own TestNodeUpdateMessage under the same TestNode.Uid, tagged with a new platform property:

public sealed class RetryAttemptProperty : IProperty
{
    public RetryAttemptProperty(int attemptNumber, bool isSuperseded);
    public int AttemptNumber { get; }   // 1-based, within a single test-host run
    public bool IsSuperseded { get; }   // a later attempt for the same uid follows
}

IsSuperseded is what lets each consumer opt in or out without guessing: consumers that want one row per test skip superseded attempts, consumers that want the history keep them all.

Consumer audit

I checked every TestNodeUpdateMessage consumer for what happens when two completed updates share a uid:

Consumer	Before	Action
`TestApplicationResult` (exit code)	counted every failed state — a retried-then-passing test would have failed the build	skip superseded
`TrxReportEngine`	duplicate `<UnitTestResult>` rows for one `<TestDefinition>`	skip superseded
`JUnitReportEngine`	kept all, renamed `[attempt N]`, inflated totals	skip superseded
`AzureDevOpsTestResultsPublisher` / reporter	duplicate rows + duplicate error annotations	skip superseded
`CtrfReportEngine`	already collapses per uid	no change — this is what fixes `flaky`
`HtmlReportEngine`	already annotates `attemptIndex` / `attemptOf`	no change
`PerRequestServerDataConsumerService`	already tracks `TotalPassedRetries` / `TotalFailedRetries`	no change
`DotnetTestDataConsumer`, `GitHubActionsReporter`	forwards / no-op	no change
Terminal (`TestProgressState`)	would double count	attempt is now the pair `(host attempt, in-process attempt)`
VSTest recorder	n/a	drops superseded, so Test Explorer / VSTest TRX are byte-identical to before

The (host attempt, in-process attempt) pair, compared lexicographically, is what makes this compose with the out-of-process --retry-failed-tests orchestrator rather than collide with it — one of the open questions in the issue.

Result

[Retry(3)] on a test that fails once and then passes:

failed (try 1) FlakyTest (10s 000ms)
  Tests failed
passed (try 2) FlakyTest (10s 000ms)

Test run summary: Passed! - assembly.dll (net8.0|x64)
  total: 1 (+1 retried)
  failed: 0
  succeeded: 1
  skipped: 0

and the CTRF report now emits "flaky": 1, "retries": 1 and a populated retryAttempts[].

Open design questions (why this is a draft)

TRX / JUnit shape — the issue flags this as the main risk. I chose to filter superseded attempts so those reports keep exactly one row per test and their counters stay correct. The alternative (emit every attempt, disambiguated) is a one-line change in JUnitReportGenerator.TryCapture / TrxDataConsumer if we'd rather expose the history there too.
(try N) on the first attempt — currently rendered for every attempt of a retry sequence, including attempt 1. A test that passes first try is untouched.
Public vs internal RetryAttemptProperty — made public so third-party frameworks can produce it and third-party reporters consume it. Easy to demote to internal if we'd rather not commit to the shape yet.
--retry-failed-tests annotation — when both retry mechanisms are active, the per-test line shows the in-process attempt. A composite try H.R rendering would need a new localized resource.
Progress counting — NotifyTestCompleted() now fires once per attempt, so the progress bar's completed count can exceed the discovered count during a retry-heavy run. Not addressed here.

Validation

Microsoft.Testing.Platform.UnitTests: 1911/1911 pass (2 new)
MSTestAdapter.PlatformServices.UnitTests: 1036/1036 pass
TestFramework.UnitTests: 1490/1490 pass
Acceptance (after -pack): RetryTests (2), CtrfReport* + JUnitReport* (6, 1 new), Trx* (6), platform *Retry* (32) — all pass

Fixes #10292

`UnitTestRunner` kept only the last retry attempt, so an in-process `[Retry]` was invisible everywhere: the terminal showed a plain passing test and the CTRF report reported `flaky: 0` with no `retryAttempts[]`, even though `CtrfReportEngine` already implements the correct rule. Every attempt is now published as its own `TestNodeUpdateMessage` under the same test node uid, tagged with a new `RetryAttemptProperty` (`AttemptNumber` + `IsSuperseded`): - Terminal: attempts are annotated (`failed (try 1) MyTest`), the test is counted once and the superseded failures feed the `(+N retried)` summary. `TestProgressState` now orders attempts by the pair (host attempt, in-process attempt) so this composes with `--retry-failed-tests` instead of colliding with it. - `TestApplicationResult`: superseded attempts are not counted, so a retried-then-passing test no longer fails the build. - TRX / JUnit / Azure DevOps: superseded attempts are filtered out, so those reports keep exactly one row per test. - CTRF and HTML: unchanged, they already collapse/annotate per uid. - VSTest: the recorder drops superseded attempts, so Test Explorer and the VSTest TRX behave exactly as before. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fa37ea7e-23a2-458b-bb8a-68ad56c832e6

Copilot

Pull request overview

Surfaces every MSTest [Retry] attempt to Microsoft.Testing.Platform, enabling terminal retry reporting and CTRF flaky-test detection.

Changes:

Adds RetryAttemptProperty and propagates retry metadata.
Reconciles attempts in terminal output and filters superseded results from selected consumers.
Adds unit, acceptance, API, RFC, and changelog updates.

Show a summary per file

File	Description
`test/UnitTests/Microsoft.Testing.Platform.UnitTests/Services/TestApplicationResultTests.cs`	Tests exit-code filtering.
`test/UnitTests/Microsoft.Testing.Platform.UnitTests/OutputDevice/Terminal/TerminalTestReporterTests.cs`	Tests retry rendering and totals.
`test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RetryTests.cs`	Validates retry terminal output.
`test/IntegrationTests/MSTest.Acceptance.IntegrationTests/JUnitReportTests.cs`	Verifies final-only JUnit results.
`test/IntegrationTests/MSTest.Acceptance.IntegrationTests/CtrfReportTests.cs`	Verifies CTRF flaky reporting.
`test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceAssert.cs`	Supports retried summary assertions.
`src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt`	Tracks new internal result properties.
`src/TestFramework/TestFramework/Attributes/TestMethod/TestResult.cs`	Stores retry attribution.
`src/TestFramework/TestFramework/Attributes/TestMethod/RetryResult.cs`	Documents attempt publication.
`src/TestFramework/TestFramework/Attributes/TestMethod/RetryBaseAttribute.cs`	Updates retry contract documentation.
`src/TestFramework/TestFramework/Attributes/TestMethod/RetryAttribute.cs`	Updates retry behavior documentation.
`src/Platform/Microsoft.Testing.Platform/Services/TestApplicationResult.cs`	Ignores superseded outcomes.
`src/Platform/Microsoft.Testing.Platform/PublicAPI/PublicAPI.Unshipped.txt`	Declares the new public property.
`src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.DataConsumption.cs`	Passes retry metadata to terminal reporting.
`src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressState.cs`	Reconciles host and in-process attempts.
`src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.TestCompletion.cs`	Annotates and counts attempts.
`src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Summary.cs`	Documents retried totals.
`src/Platform/Microsoft.Testing.Platform/Messages/RetryAttemptProperty.cs`	Defines retry attempt metadata.
`src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxDataConsumer.cs`	Filters superseded TRX results.
`src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportGenerator.cs`	Filters superseded JUnit results.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs`	Filters superseded published results.
`src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs`	Suppresses superseded annotations.
`src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs`	Flattens and tags retry attempts.
`src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs`	Uses final outcomes for verdicts.
`src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestTestNodeConverter.cs`	Adds retry properties to test nodes.
`src/Adapter/MSTest.TestAdapter/Services/TestResultRecorderExtensions.cs`	Preserves VSTest final-only behavior.
`docs/RFCs/016-JUnit-Report.md`	Documents JUnit retry handling.
`docs/Changelog.md`	Records the MSTest behavior change.
`docs/Changelog-Platform.md`	Records the platform API change.

Review details

Comments suppressed due to low confidence (1)

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:357

A custom RetryBaseAttribute may legally add an empty final result array. Previously TryGetLast() returned that empty array, allowing SendTestResultsAsync to call RecordEmptyResultAsync; this combination instead retains only earlier superseded results, so no final outcome is recorded and AllPassed can return true after skipping them. Preserve the existing empty-result behavior when the final retry attempt is empty.

        IReadOnlyList<TestResult[]> retryAttempts = retryResult.AllResults;
        if (retryAttempts.Count == 0)
        {
            // A RetryBaseAttribute implementation is expected to add at least one attempt.
            throw ApplicationStateGuard.Unreachable();

Files reviewed: 29/29 changed files
Comments generated: 7
Review effort level: Medium

+    public void ReportPassingTest(string testNodeUid, string instanceId, int retryAttemptNumber)
+        => ReportGenericTestResult(testNodeUid, instanceId, retryAttemptNumber, static entry => entry with { Passed = entry.Passed + 1 }, static @this => @this._passedTests++);
+
+    public void ReportSkippedTest(string testNodeUid, string instanceId, int retryAttemptNumber)
+        => ReportGenericTestResult(testNodeUid, instanceId, retryAttemptNumber, static entry => entry with { Skipped = entry.Skipped + 1 }, static @this => @this._skippedTests++);


+            // The orchestrator attributes retries per host instance; it does not surface a test framework's
+            // in-process retry attempt, so results arriving through this path are always attempt 1 of their host
+            // attempt.
+            retryAttemptNumber: 1,
+            isRetryAttempt: false);


+    /// The final attempt stays last so callers that look at <c>results[^1]</c> (class and assembly cleanup) and
+    /// consumers that collapse per test node uid keep observing the test's real outcome.


+        if (result.RetryAttemptNumber > 1 || result.IsSupersededRetryAttempt)
+        {
+            testNode.Properties.Add(new RetryAttemptProperty(result.RetryAttemptNumber, result.IsSupersededRetryAttempt));


@@ -1,4 +1,4 @@
-// Copyright (c) Microsoft Corporation. All rights reserved.
+// Copyright (c) Microsoft Corporation. All rights reserved.


@@ -0,0 +1,72 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


+                asm.ReportFailedTest(testNodeUid, instanceId, retryAttemptNumber);
                break;
            case TestOutcome.Passed:
-                asm.ReportPassingTest(testNodeUid, instanceId);
+                asm.ReportPassingTest(testNodeUid, instanceId, retryAttemptNumber);
                break;
            case TestOutcome.Skipped:
-                asm.ReportSkippedTest(testNodeUid, instanceId);
+                asm.ReportSkippedTest(testNodeUid, instanceId, retryAttemptNumber);


Copilot AI review requested due to automatic review settings July 28, 2026 18:11

Copilot started reviewing on behalf of Evangelink July 28, 2026 18:12 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Report MSTest `[Retry]` attempts to the platform instead of discarding them - #10295

Report MSTest `[Retry]` attempts to the platform instead of discarding them#10295
Evangelink wants to merge 1 commit into
mainfrom
dev/amauryleve/literate-lamp

Evangelink commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		/// The final attempt stays last so callers that look at <c>results[^1]</c> (class and assembly cleanup) and
		/// consumers that collapse per test node uid keep observing the test's real outcome.

		@@ -1,4 +1,4 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,72 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

Uh oh!

Conversation

Evangelink commented Jul 28, 2026

Problem

Approach

Consumer audit

Result

Open design questions (why this is a draft)

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants