CI: desktop source enumeration does not return on ~2 of 5 attempts on GitHub runners

[EtienneLescot]

nix-build.yml's smoke test cannot reliably go green, and it is the last open question from the nix packaging work (#419, #429, #436, #438, #447, #448, #449, #455, #458 — all merged). Everything else in that series is built, wired and asserted; this is not.

Filing it so the packaging work can close without the flakiness closing with it.

The phenomenon

desktopCapturer.getSources on a GitHub runner under xvfb-run either answers almost instantly or never returns. Both CLI paths hit it: openscreen sources (bounded at 20s in the renderer) and openscreen record (bounded at 30s in the shared get-sources handler).

Rate is roughly 2 failures in 5 attempts, but that average is misleading — see below.

It is bimodal, not slow

The [get-sources] duration added in #436 is the useful part. Across five runs, every successful call:

12ms  13ms  14ms  16ms  16ms  17ms  17ms  18ms  18ms  19ms  19ms  31ms

and every failure returns nothing at all — no [get-sources] line is emitted, because the call never completes. There is no intermediate value in any measurement taken.

This matters practically: there is no threshold to tune. The bounds are 20s and 30s against a call that takes 17ms when it works, so raising or lowering them changes nothing.

The failures cluster; they are not independent

Per-run, with run_cli invocation index (added in #438):

run	sources	record
32408517738	2/5	0/3
32422772587	4/5	0/3
32475812049	3/5	0/3
32479979866	4/5	1/1 ✓
32480776100	3/5	1/1 ✓
32494267575	2/5	1/1 ✓
32535931265	3/5 (fail at #4, #5; ok at #6, #7, #8)	0/3 (fail at #9)

Two things follow, and both were mistakes made and corrected along the way:

1. There is no "record is special". An early reading had record at 1/10 against sources' 13/20 and concluded something about the record path. That gap was per-run clustering seen through a denominator; record has since succeeded first-try from both its old and new positions.
2. There is no ordering effect that the data supports. ci(nix): run record before sources, to test whether position is the cause #438 moved the record block ahead of the sources loop to test whether invocation position mattered; ci(nix): put record back after sources, and record why the experiment failed #447 moved it back. The experiment could not answer — by the time it ran, record was already succeeding from its old position, so there was no contrast left. The last run is the clearest evidence against position: failures at invocations 4, 5 and 9 with three successes at 6, 7, 8 in between. Position would predict monotonic decay with rank, not a hole at each end.

The shape that fits is a time window within a run during which the capture stack is unavailable, independent of which command asks.

What is already instrumented

No new tooling is needed to investigate:

• [get-sources] returned N source(s) in Xms / failed after Xms: <reason> on both paths, under OPENSCREEN_DIAGNOSTIC=1 (electron/ipc/handlers.ts).
• Every attempt prints its run_cli invocation index, and the verdict prints the total, so rates can be stratified by position across runs rather than averaged across it.
• OPENSCREEN_DIAGNOSTIC startup milestones show how far the main process got before the renderer went silent.

What is not instrumented is anything below the Electron API — no Xvfb log is captured, and nothing observes the X11 side of the failed call.

Why it costs something

The job gates on EXPORT_OK, which needs record to succeed at least once in three tries. So a run is green or red on runner luck rather than on the derivation, which is the opposite of what the check is for. Two of the last seven runs were green.

Enumeration flakiness itself is already demoted to a ::warning:: (#429) and does not fail the job; only the export gate does.

What would move it

In rough order of cost:

1. Capture the Xvfb server log alongside the CLI output. Nothing currently observes the X11 side, and the failure is almost certainly there.
2. Check whether the failed calls correspond to a specific xvfb-run -a display allocation — run_cli spawns a fresh Xvfb per invocation, fourteen per job.
3. Reproduce outside CI on a headless container, which would make the loop minutes rather than half an hour.

Probably unrelated, but worth ruling out rather than assuming: #324 is a Wayland/niri capture failure in released builds. This one is X11 under Xvfb on a CI runner.