ci: bound the Linux apt install so a stalled mirror fails fast - #1887
Merged
Conversation
Unbounded, the desktop-dependency install could not fail, only stall. On 2026-08-19 a slow package mirror held apt past the job's 30-minute budget on four main-branch runs and several unrelated PRs, cancelling each job before Setup toolchain, Xvfb/D-Bus, or the replay smoke test ran — a red check on branches that never executed a line of project code. timeout-minutes: 6 turns that into a named step failure in six minutes instead of a cancelled job at thirty; a healthy install takes about a minute. The apt options cover the transient cases without a retry loop layered on top of them: socket timeouts bound a mirror that connects and then goes quiet, Acquire::Retries absorbs a blip, and DPkg::Lock::Timeout bounds the runner's own unattended-upgrades timer, which stalls identically and is a plausible alternate cause of the same symptom. Tradeoff: a fast transient failure that apt's own retries miss now fails the job rather than self-healing, traded against carrying a bash retry loop in CI.
Size Report
Startup median (7 runs, lower is better):
Top changed chunks:
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
Install Linux desktop dependenciesstep in.github/workflows/linux.ymlhad no bound of its own, so it could not fail — only stall.On 2026-08-19, PR #1856 run 32268264207 was cancelled at this step on both attempts (30m24s each). Everything downstream — Setup toolchain, Start Xvfb and D-Bus, Run Linux replay smoke test — reported as skipped. In the same window, three Linux runs on unrelated branches (
agent/wave4-get,test/1781-b1-daemon-leak-oracle,claude/1832-c3-residues) were stuck at the identical step and four main-branch runs were cancelled. Environmental, not branch-specific.The result is a red check on branches that never executed a line of project code, and a 30-minute runner burn per attempt.
Change
Two bounds, 18 lines, package list untouched:
timeout-minutes: 6— a stall now surfaces as a named step failure in six minutes instead of a cancelled job at thirty, leaving the job 24 of its 30 minutes for the smoke test it exists to run. A healthy install takes about a minute.Acquire::http[s]::Timeout=15bounds a mirror that connects and then goes quiet,Acquire::Retries=2absorbs a transient blip, andDPkg::Lock::Timeout=60bounds the runner's ownunattended-upgradestimer, which stalls identically and is a plausible alternate cause of the same symptom.Notes on what this deliberately does not do
An earlier draft added a bash retry loop with per-attempt
timeoutwrappers and anarchive.ubuntu.commirror fallback (~80 lines). Dropped:Acquire::Retriesalready retries inside apt, thetimeoutwrappers existed only to make the outer loop work during a stall, and the mirror fallback was ~25 lines ofsedacross two sources formats to salvage a job that now just fails clearly. It also can't be made unconditional —archive.ubuntu.comis slower from Azure-hosted runners.Caching the apt packages or baking them into a prebuilt image would remove the network from the hot path entirely, but caching
.debs still needsapt-get updatefor dependency resolution, so only the image approach actually delivers that — and that's a CI-infrastructure change, not a hardening of this step. Worth a separate issue if stalls recur.Tradeoff: a fast transient failure that apt's own retries miss now fails the job rather than self-healing, traded against carrying a bash retry loop in CI.
Verification
Locally: YAML parses,
shellcheck -s bashclean on the extracted script, and theAPT_OPTSarray verified to expand onto both apt calls with all 10 packages intact.The 6-minute fast-fail can only be confirmed by a real run, since a stall depends on mirror conditions that aren't reproducible locally. This PR's own Linux job exercises the happy path.