Skip to content

Commit 5384069

Browse files
KSemenenkoKSemenenko
committed
Stabilize browser suites under parallel load
1 parent b6eaeaa commit 5384069

29 files changed

Lines changed: 281 additions & 128 deletions

‎stabilize-browser-suite-parallelism.plan.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Stabilize Browser Suite Parallelism
2+
3+
## Goal
4+
5+
Make the browser acceptance suites deterministic under the higher in-suite parallelism the repo already allows, then prove the fix with full local baseline runs and the replacement GitHub Actions run on `main`.
6+
7+
## Scope
8+
9+
In scope:
10+
- `tests/PrompterOne.Web.UITests/Infrastructure/*` and `tests/PrompterOne.Web.UITests/Support/*` shared browser harness helpers
11+
- `tests/PrompterOne.Web.UITests.Shell/**/*`
12+
- `tests/PrompterOne.Web.UITests.Studio/**/*`
13+
- `tests/PrompterOne.Web.UITests.Reader/**/*`
14+
- `tests/PrompterOne.Web.UITests.Editor/**/*` where the shared-harness fix or route-wait fix changes expectations
15+
- Push to `main` and watch the replacement Actions run
16+
17+
Out of scope:
18+
- Lowering CI parallelism back to `2`
19+
- Timeout-only “fixes”
20+
- Unrelated product behavior changes outside what is required for test determinism
21+
22+
## Constraints And Risks
23+
24+
- Keep one browser-suite `dotnet test` process at a time locally.
25+
- Do not revert the user’s requested higher CI concurrency posture.
26+
- Prefer harness isolation and stable route contracts over weaker assertions.
27+
- Shared-context tests that intentionally validate storage/session propagation must stay shared.
28+
- Preserve production-owned `data-test` hooks and route contracts.
29+
30+
## Testing Methodology
31+
32+
- Establish the failing baseline from the real browser suites and GitHub Actions logs first.
33+
- Fix one failure class at a time: shared `BrowserContext` teardown interference, then SPA route-wait flakiness.
34+
- Verify by layers:
35+
- targeted previously failing suites
36+
- all changed browser suites
37+
- full `dotnet test --solution ./PrompterOne.slnx --max-parallel-test-modules 1`
38+
- After local green, run `dotnet format`, push `main`, and monitor the new Actions run until green or a concrete external blocker remains.
39+
40+
## Baseline And Failing Tests
41+
42+
- [x] Baseline local browser-suite failures reproduced and correlated with the user-reported red tests.
43+
- [x] Previous failed GitHub Actions run inspected.
44+
Result:
45+
- Shell, Studio, and Reader had widespread first-screen visibility failures under parallel load.
46+
- Solution-level rerun exposed an additional SPA-route timing failure in `GoLiveFlowTests.SettingsPage_LinksIntoGoLiveRoutingAndGoLiveLinksBackToSettings`.
47+
48+
Tracked failures:
49+
- [x] `EditorAiAvailabilityTests.EditorScreen_AiButtonsAreEnabled_WhenAProviderIsConfigured`
50+
Symptom: `editor-page` not found.
51+
Root-cause note: direct editor tests reused shared contexts and then closed them from sibling tests.
52+
Intended fix path: force isolated contexts for direct page-based editor tests.
53+
- [x] `EditorCueRenderingFlowTests.EditorScreen_RendersMonacoCueStylesImmediatelyAfterImport`
54+
Symptom: `editor-page` / `editor-source-stage` not found.
55+
Root-cause note: same shared-context teardown interference.
56+
Intended fix path: isolate direct editor tests.
57+
- [x] `EditorFindFlowTests.EditorScreen_FindBar_UsesStyledChrome`
58+
Symptom: `editor-source-stage` not found.
59+
Root-cause note: same shared-context teardown interference.
60+
Intended fix path: isolate direct editor tests.
61+
- [x] `LibraryScreen_OpenScriptImportsLocalFileAndNavigatesIntoEditor`
62+
Symptom: wrong imported title shown in header.
63+
Root-cause note: already fixed in prior import-title pass; kept under regression watch here.
64+
Intended fix path: no new action unless the broader stabilization regresses it.
65+
- [x] `LibraryScreen_OpenScriptCanImportASecondFile_AfterPickerResets`
66+
Symptom: wrong imported title shown in header.
67+
Root-cause note: already fixed in prior import-title pass; kept under regression watch here.
68+
Intended fix path: no new action unless the broader stabilization regresses it.
69+
- [x] `GoLiveFlowTests.SettingsPage_LinksIntoGoLiveRoutingAndGoLiveLinksBackToSettings`
70+
Symptom: route wait timed out on `**/settings` during SPA navigation.
71+
Root-cause note: `WaitForURLAsync` depended on navigation/load semantics that were race-prone for in-app route changes under suite load.
72+
Intended fix path: replace navigation-event waits with an assertion-based SPA route helper.
73+
74+
## Ordered Plan
75+
76+
- [x] Step 1. Inspect the shared browser harness and identify which tests intentionally share contexts versus accidentally reuse them.
77+
Verification:
78+
- Confirmed `StandaloneAppFixture.NewPageAsync()` defaults to shared contexts keyed by caller member name.
79+
- Confirmed many direct page-based tests in Editor, Shell, Studio, and Reader were not intended to share contexts.
80+
81+
- [x] Step 2. Establish the red baseline locally and from CI logs.
82+
Verification:
83+
- Reproduced local editor and browser-suite failures.
84+
- Pulled CI job logs for the failed `main` run and mapped the failure pattern.
85+
86+
- [x] Step 3. Isolate direct page-based tests that were closing shared contexts.
87+
Actions:
88+
- Converted direct `NewPageAsync()` callers in the affected suites to `NewPageAsync(additionalContext: true)`.
89+
- Preserved the intentionally shared tests in `DynamicHostPortTests` and the existing `NewSharedPagesAsync(...)` cases.
90+
Verification:
91+
- `PrompterOne.Web.UITests.Editor` passed under `--maximum-parallel-tests 8`.
92+
- Shell, Studio, and Reader targeted reruns cleared the widespread first-screen visibility failures.
93+
94+
- [x] Step 4. Replace brittle SPA route waits with a deterministic helper.
95+
Actions:
96+
- Added `tests/PrompterOne.Web.UITests/Support/BrowserRouteDriver.cs`.
97+
- Swapped `WaitForURLAsync(BrowserTestConstants.Routes.Pattern(...))` for `BrowserRouteDriver.WaitForRouteAsync(...)` across the affected browser suites, leaving the one variable-pattern onboarding wait unchanged.
98+
Verification:
99+
- The previously failing `GoLiveFlowTests.SettingsPage_LinksIntoGoLiveRoutingAndGoLiveLinksBackToSettings` now passes.
100+
- Updated Shell, Studio, Reader, and Editor suites build and run successfully.
101+
102+
- [x] Step 5. Run changed-suite verification.
103+
Verification:
104+
- `dotnet build ./PrompterOne.slnx -warnaserror` passed.
105+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Studio/PrompterOne.Web.UITests.Studio.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20` passed.
106+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Shell/PrompterOne.Web.UITests.Shell.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20` passed.
107+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Reader/PrompterOne.Web.UITests.Reader.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20` passed.
108+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Editor/PrompterOne.Web.UITests.Editor.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20` passed.
109+
110+
- [x] Step 6. Run the required full-solution baseline.
111+
Command:
112+
- `dotnet test @./tests/dotnet-test-progress.rsp --solution ./PrompterOne.slnx --max-parallel-test-modules 1`
113+
Verification:
114+
- Full solution passes after the shared-harness and route-wait fixes.
115+
- Repeated after `dotnet format` and a fresh `dotnet build ./PrompterOne.slnx -warnaserror`; both full-solution reruns finished green at `1152/1152`.
116+
117+
- [ ] Step 7. Run formatting and publish the fix.
118+
Actions:
119+
- Run `dotnet format ./PrompterOne.slnx`.
120+
- Stage only task-relevant files.
121+
- Commit on `main`.
122+
- Push `main`.
123+
Verification:
124+
- Local tree is clean except for expected plan artifacts.
125+
126+
- [ ] Step 8. Monitor the replacement GitHub Actions run.
127+
Verification:
128+
- The new `main` run finishes green or any remaining blocker is documented with the specific failing job/test.
129+
130+
## Final Validation Skills And Commands
131+
132+
- `dotnet`
133+
Reason: repo-standard build and test validation for the changed .NET/browser code.
134+
- `github:gh-fix-ci`
135+
Reason: inspect and monitor the replacement GitHub Actions run on `main`.
136+
- `dotnet build ./PrompterOne.slnx -warnaserror`
137+
Reason: required repository build gate.
138+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Studio/PrompterOne.Web.UITests.Studio.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20`
139+
Reason: directly verifies the SPA route-wait fix in the suite that exposed it.
140+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Shell/PrompterOne.Web.UITests.Shell.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20`
141+
Reason: confirms the broader shell/shared-context fixes stay green.
142+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Reader/PrompterOne.Web.UITests.Reader.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20`
143+
Reason: confirms the reader suite remains stable after shared-harness changes.
144+
- `dotnet test @./tests/dotnet-test-progress.rsp --project ./tests/PrompterOne.Web.UITests.Editor/PrompterOne.Web.UITests.Editor.csproj --maximum-parallel-tests 8 --maximum-failed-tests 20`
145+
Reason: confirms editor stability after the context-isolation fix.
146+
- `dotnet test @./tests/dotnet-test-progress.rsp --solution ./PrompterOne.slnx --max-parallel-test-modules 1`
147+
Reason: required full-solution regression gate from root instructions.
148+
- `dotnet format ./PrompterOne.slnx`
149+
Reason: required repository formatting gate before finalizing the change.

‎tests/PrompterOne.Web.UITests.Editor/Editor/EditorDocumentSplitFlowTests.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ await UiScenarioArtifacts.CapturePageAsync(
5757
await Assert.That(new Uri(page.Url).AbsolutePath).IsEqualTo(AppRoutes.Editor);
5858

5959
await page.GetByTestId(UiTestIds.Editor.SplitResultOpenLibrary).ClickAsync();
60-
await page.WaitForURLAsync(BrowserTestConstants.Routes.Pattern(AppRoutes.Library));
60+
await BrowserRouteDriver.WaitForRouteAsync(page, AppRoutes.Library);
6161
await Expect(page.GetByTestId(UiTestIds.Library.Page)).ToBeVisibleAsync();
6262
await page.GetByTestId(UiTestIds.Library.FolderAll).ClickAsync();
6363

@@ -100,7 +100,7 @@ await UiScenarioArtifacts.CapturePageAsync(
100100
await Assert.That(new Uri(page.Url).AbsolutePath).IsEqualTo(AppRoutes.Editor);
101101

102102
await page.GetByTestId(UiTestIds.Editor.SplitResultOpenLibrary).ClickAsync();
103-
await page.WaitForURLAsync(BrowserTestConstants.Routes.Pattern(AppRoutes.Library));
103+
await BrowserRouteDriver.WaitForRouteAsync(page, AppRoutes.Library);
104104
await Expect(page.GetByTestId(UiTestIds.Library.Page)).ToBeVisibleAsync();
105105
await page.GetByTestId(UiTestIds.Library.FolderAll).ClickAsync();
106106

‎tests/PrompterOne.Web.UITests.Reader/Learn/EditorLearnScreenFlowTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ await Expect(page.GetByTestId(UiTestIds.Editor.SegmentNavigation(2))).ToHaveAttr
3535

3636
await Expect(page.GetByTestId(UiTestIds.Header.EditorLearn)).ToBeVisibleAsync();
3737
await page.GetByTestId(UiTestIds.Header.EditorLearn).ClickAsync();
38-
await page.WaitForURLAsync(BrowserTestConstants.Routes.Pattern(BrowserTestConstants.Routes.LearnDemo));
38+
await BrowserRouteDriver.WaitForRouteAsync(page, BrowserTestConstants.Routes.LearnDemo);
3939
await Expect(page.GetByTestId(UiTestIds.Learn.Page)).ToBeVisibleAsync(new() { Timeout = BrowserTestConstants.Timing.ExtendedVisibleTimeoutMs });
4040

4141
await page.GotoAsync(BrowserTestConstants.Routes.LearnDemo);

‎tests/PrompterOne.Web.UITests.Reader/Learn/LearnControlAlignmentTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ public sealed class LearnControlAlignmentTests(StandaloneAppFixture fixture)
1414
[Test]
1515
public async Task LearnScreen_QuantumControls_StayCenteredAndSymmetric()
1616
{
17-
var page = await _fixture.NewPageAsync();
17+
var page = await _fixture.NewPageAsync(additionalContext: true);
1818

1919
try
2020
{

‎tests/PrompterOne.Web.UITests.Reader/Learn/LearnFidelityTests.cs‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ public sealed class LearnFidelityTests(StandaloneAppFixture fixture)
1919
[Test]
2020
public async Task LearnScreen_KeepsOrpLetterCenteredOnReferenceGuide()
2121
{
22-
var page = await fixture.NewPageAsync();
22+
var page = await fixture.NewPageAsync(additionalContext: true);
2323

2424
try
2525
{
@@ -48,7 +48,7 @@ await Expect(page.GetByTestId(UiTestIds.Learn.Page))
4848
[Test]
4949
public async Task LearnScreen_UsesPhraseTimelineForSecurityIncidentScript()
5050
{
51-
var page = await fixture.NewPageAsync();
51+
var page = await fixture.NewPageAsync(additionalContext: true);
5252

5353
try
5454
{
@@ -84,7 +84,7 @@ await Expect(page.GetByTestId(UiTestIds.Learn.NextPhrase))
8484
[Test]
8585
public async Task LearnScreen_DemoContextRails_ShowTwoWordsPerSideWithoutRightRailClipping()
8686
{
87-
var page = await fixture.NewPageAsync();
87+
var page = await fixture.NewPageAsync(additionalContext: true);
8888

8989
try
9090
{
@@ -127,7 +127,7 @@ await Assert.That(rightWords).IsEquivalentTo([
127127
[Test]
128128
public async Task LearnScreen_KeepsContextRailsSeparatedFromFocusWordOnLeadershipScript()
129129
{
130-
var page = await fixture.NewPageAsync();
130+
var page = await fixture.NewPageAsync(additionalContext: true);
131131

132132
try
133133
{
@@ -155,7 +155,7 @@ await Expect(page.GetByTestId(UiTestIds.Learn.Page))
155155
[Test]
156156
public async Task LearnScreen_KeepsSecurityIncidentContextWordsCloseToFocusedWord()
157157
{
158-
var page = await fixture.NewPageAsync();
158+
var page = await fixture.NewPageAsync(additionalContext: true);
159159

160160
try
161161
{
@@ -186,7 +186,7 @@ await StepUntilWordAsync(
186186
[Test]
187187
public async Task LearnScreen_KeepsQuantumContextWordsCloseToFocusedWord()
188188
{
189-
var page = await fixture.NewPageAsync();
189+
var page = await fixture.NewPageAsync(additionalContext: true);
190190

191191
try
192192
{
@@ -217,7 +217,7 @@ await StepUntilWordAsync(
217217
[Test]
218218
public async Task LearnScreen_LeadershipPreviewState_ShowsCurrentSentenceContextAndCloserContext()
219219
{
220-
var page = await fixture.NewPageAsync();
220+
var page = await fixture.NewPageAsync(additionalContext: true);
221221

222222
try
223223
{
@@ -251,7 +251,7 @@ await Expect(page.GetByTestId(UiTestIds.Learn.NextPhrase))
251251
[Test]
252252
public async Task LearnScreen_LeadershipUncertainState_StaysSentenceLocalAndDropsPunctuation()
253253
{
254-
var page = await fixture.NewPageAsync();
254+
var page = await fixture.NewPageAsync(additionalContext: true);
255255

256256
try
257257
{
@@ -296,7 +296,7 @@ await Expect(page.GetByTestId(UiTestIds.Learn.NextPhrase))
296296
[Test]
297297
public async Task LearnScreen_LongFocusWord_FitsWithoutClipping()
298298
{
299-
var page = await fixture.NewPageAsync();
299+
var page = await fixture.NewPageAsync(additionalContext: true);
300300

301301
try
302302
{
@@ -323,7 +323,7 @@ await StepUntilWordAsync(
323323
[Test]
324324
public async Task LearnScreen_FocusWord_UsesPackedHorizontalStackAroundOrp()
325325
{
326-
var page = await fixture.NewPageAsync();
326+
var page = await fixture.NewPageAsync(additionalContext: true);
327327

328328
try
329329
{
@@ -496,7 +496,7 @@ private static async Task DecreaseLearnSpeedAsync(Microsoft.Playwright.IPage pag
496496

497497
private async Task<int> MeasureLearnPlaybackDurationAsync(Func<Microsoft.Playwright.IPage, Task> configurePageAsync)
498498
{
499-
var page = await fixture.NewPageAsync();
499+
var page = await fixture.NewPageAsync(additionalContext: true);
500500

501501
try
502502
{

‎tests/PrompterOne.Web.UITests.Reader/Learn/LearnStartupAlignmentTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ public sealed class LearnStartupAlignmentTests(StandaloneAppFixture fixture)
2020
[Test]
2121
public async Task LearnScreen_DemoStartup_HidesFocusRowUntilOrpLayoutIsReady()
2222
{
23-
var page = await _fixture.NewPageAsync();
23+
var page = await _fixture.NewPageAsync(additionalContext: true);
2424

2525
try
2626
{

‎tests/PrompterOne.Web.UITests.Reader/Learn/LearnWordLaneStabilityTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ public sealed class LearnWordLaneStabilityTests(StandaloneAppFixture fixture)
1717
[Test]
1818
public async Task LearnScreen_QuantumWordLengthChanges_KeepTheOrpAnchorAndVisibleContextGapsStable()
1919
{
20-
var page = await _fixture.NewPageAsync();
20+
var page = await _fixture.NewPageAsync(additionalContext: true);
2121

2222
try
2323
{

‎tests/PrompterOne.Web.UITests.Reader/Teleprompter/TeleprompterCueRenderingFlowTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ public async Task TeleprompterDemo_RendersTypographyDrivenCueVariablesForVolumeA
1515
{
1616
UiScenarioArtifacts.ResetScenario(CueScenario);
1717

18-
var page = await fixture.NewPageAsync();
18+
var page = await fixture.NewPageAsync(additionalContext: true);
1919

2020
try
2121
{

‎tests/PrompterOne.Web.UITests.Reader/Teleprompter/TeleprompterFidelityTests.cs‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ public sealed class TeleprompterFidelityTests(StandaloneAppFixture fixture)
2121
[Test]
2222
public async Task TeleprompterLeadership_RepositionsReadingLineWhenFocalPointChanges()
2323
{
24-
var page = await fixture.NewPageAsync();
24+
var page = await fixture.NewPageAsync(additionalContext: true);
2525

2626
try
2727
{
@@ -55,7 +55,7 @@ await page.GetByTestId(UiTestIds.Teleprompter.FocalSlider).EvaluateAsync(
5555
[Test]
5656
public async Task TeleprompterScreen_UsesSingleFullBleedBackgroundCameraLayer()
5757
{
58-
var page = await fixture.NewPageAsync();
58+
var page = await fixture.NewPageAsync(additionalContext: true);
5959

6060
try
6161
{
@@ -97,7 +97,7 @@ await Expect(page.GetByTestId(UiTestIds.Teleprompter.Page))
9797
[Test]
9898
public async Task TeleprompterDemo_KeepsParagraphStableWhenFirstWordsAdvance()
9999
{
100-
var page = await fixture.NewPageAsync();
100+
var page = await fixture.NewPageAsync(additionalContext: true);
101101

102102
try
103103
{
@@ -121,7 +121,7 @@ await Expect(page.GetByTestId(UiTestIds.Teleprompter.Page))
121121
[Test]
122122
public async Task TeleprompterDemo_KeepsParagraphStableWhenFontSizeChanges()
123123
{
124-
var page = await fixture.NewPageAsync();
124+
var page = await fixture.NewPageAsync(additionalContext: true);
125125

126126
try
127127
{
@@ -144,7 +144,7 @@ await Expect(page.GetByTestId(UiTestIds.Teleprompter.Page))
144144
[Test]
145145
public async Task TeleprompterDemo_ActivatesNextWordDirectlyOnFocalGuideWithoutVisibleSettling()
146146
{
147-
var page = await fixture.NewPageAsync();
147+
var page = await fixture.NewPageAsync(additionalContext: true);
148148

149149
try
150150
{
@@ -177,7 +177,7 @@ await Expect(page.GetByTestId(UiTestIds.Teleprompter.Page))
177177
[Test]
178178
public async Task TeleprompterSecurityIncident_UsesMaximumWidthAndContinuousEmphasisWithoutStandaloneCommaWords()
179179
{
180-
var page = await fixture.NewPageAsync();
180+
var page = await fixture.NewPageAsync(additionalContext: true);
181181

182182
try
183183
{

‎tests/PrompterOne.Web.UITests.Reader/Teleprompter/TeleprompterFullFlowTests.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ public sealed class TeleprompterFullFlowTests(StandaloneAppFixture fixture)
1818
[Test]
1919
public async Task TeleprompterProductLaunch_FullTpsScenario_CapturesArtifactsAndKeepsAlignment()
2020
{
21-
var page = await fixture.NewPageAsync();
21+
var page = await fixture.NewPageAsync(additionalContext: true);
2222

2323
try
2424
{
@@ -62,7 +62,7 @@ await UiScenarioArtifacts.CapturePageAsync(
6262
[Test]
6363
public async Task TeleprompterSpeedOffsets_UsesCustomFrontMatterSpeedOffsetsAndNormalResetSpacing()
6464
{
65-
var page = await fixture.NewPageAsync();
65+
var page = await fixture.NewPageAsync(additionalContext: true);
6666

6767
try
6868
{

0 commit comments

Comments
 (0)