Skip to content

CI: add timeout and retry to apt-get install steps - #624

Closed
mmcky wants to merge 3 commits into
mainfrom
ci/apt-timeout-retry
Closed

CI: add timeout and retry to apt-get install steps#624
mmcky wants to merge 3 commits into
mainfrom
ci/apt-timeout-retry

Conversation

@mmcky

@mmcky mmcky commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Bare sudo apt-get install -y with no timeout-minutes and no retry lets a stalled Ubuntu mirror hang CI for hours: two 358-minute hangs on 2026-08-18 across two edition repos ten minutes apart (a mirror outage), four more hangs on 2026-08-19 (Install latex dependencies 77/15/16 min, Graphics Support 21 min) each cleared only by cancel-and-rerun, and a 59-minute slow-but-green install on 2026-08-20.

This PR hardens every apt-installing step in the workflows that carry one (ci.yml, cache.yml, publish.yml where present):

  • a step-level timeout-minutes (10 for graphviz, 30 for the texlive set), so a hang can no longer take the six-hour job limit;
  • up to three attempts with a per-attempt timeout (3m / 9m) and a 30-second pause, so a transient mirror outage is ridden out rather than failing the build — a timeout alone would only turn a six-hour hang into a fast red build.

Package lists are unchanged. The apt steps are edition-local (the shared QuantEcon/actions/templates carry none), so this lands per repo. Part of the five-repo sweep tracked in QuantEcon/project-translation#43.

Bare `sudo apt-get install -y` with no `timeout-minutes` and no retry let a stalled Ubuntu mirror hang CI for hours — two 358-minute hangs on 2026-08-18 across two edition repos ten minutes apart, and four more hangs on 2026-08-19 each cleared only by cancel-and-rerun.

Every apt-installing step now carries a step-level `timeout-minutes` and runs up to three attempts with a per-attempt `timeout`, so a stalled fetch fails fast and a transient mirror outage is ridden out rather than burning the job. Package lists are unchanged.

Tracked in QuantEcon/project-translation#43.
Copilot AI lite review requested due to automatic review settings August 20, 2026 23:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens GitHub Actions workflows against Ubuntu mirror stalls by adding step-level timeouts and retry logic around the LaTeX dependency installation steps, reducing the chance of multi-hour CI hangs in this Jupyter Book-based lecture build/publish pipeline.

Changes:

  • Add timeout-minutes to the “Install latex dependencies” steps.
  • Wrap apt-get update + apt-get install in a 3-attempt retry loop with per-attempt timeouts and a short backoff.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/ci.yml Adds step timeout and retry/timeout loop around LaTeX apt installs in PR CI.
.github/workflows/publish.yml Adds step timeout and retry/timeout loop around LaTeX apt installs in publish workflow.
Suppressed comments (2)

.github/workflows/ci.yml:46

  • The retry loop uses a single timeout 9m covering both apt-get update and apt-get install, but the PR description says the per-attempt timeouts are split (3m / 9m). As written, a slow update consumes install budget and also doesn’t match the stated behavior; splitting the timeouts makes the step’s intent clearer and more predictable.
          for attempt in 1 2 3; do
            timeout 9m sudo bash -c "apt-get -qq update && apt-get install -y $pkgs" && break
            if [ "$attempt" = 3 ]; then echo "apt-get failed after 3 attempts"; exit 1; fi

.github/workflows/publish.yml:62

  • The retry loop uses a single timeout 9m covering both apt-get update and apt-get install, but the PR description says the per-attempt timeouts are split (3m / 9m). As written, a slow update consumes install budget and also doesn’t match the stated behavior; splitting the timeouts makes the step’s intent clearer and more predictable.
          for attempt in 1 2 3; do
            timeout 9m sudo bash -c "apt-get -qq update && apt-get install -y $pkgs" && break
            if [ "$attempt" = 3 ]; then echo "apt-get failed after 3 attempts"; exit 1; fi

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 54 to 56
- name: Install latex dependencies
timeout-minutes: 30
run: |
Comment thread .github/workflows/ci.yml
Comment on lines 38 to 40
- name: Install latex dependencies
timeout-minutes: 30
run: |
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

mmcky added 2 commits August 21, 2026 09:40
…er-attempt timeouts (4m/14m)

Addresses Copilot review on QuantEcon/lecture-python.zh-cn#263: the cache.yml graphviz step was missed by a name match, and three 3m/9m attempts could never let a slow-but-progressing install finish. Now two attempts at 4m (graphviz) / 14m (texlive) inside the unchanged 10/30-minute step budgets.
…-get

Addresses Copilot review on QuantEcon/lecture-intro.zh-cn#301: with timeout wrapping sudo, a timed-out attempt terminated sudo and the bash -c shell but could orphan apt-get holding the dpkg lock, making the retry block instead of retry. timeout is now the direct parent of each apt-get (inside sudo, -k 30s escalation): update 1m/2m, install 3m/12m for graphviz/texlive. Step budgets unchanged.
@mmcky

mmcky commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Closing unmerged (@mmcky, 2026-08-21): the apt hangs are accepted as a known flake for now rather than investing further here — the QuantEcon/actions container templates remove the apt step entirely, and that deployment is the medium-term fix. Branch left in place for reference; tracked in QuantEcon/project-translation#43.

@mmcky mmcky closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants