ci: unified workflow with build-once image sharing and cross-run layer cache#1145
Open
dimitri wants to merge 2 commits into
Open
ci: unified workflow with build-once image sharing and cross-run layer cache#1145dimitri wants to merge 2 commits into
dimitri wants to merge 2 commits into
Conversation
…r cache
Replace the two independent workflows (run-tests.yml, run-pgaftest.yml)
with a single ci.yml that builds each Postgres version's images once and
shares them across both the pytest suite and the pgaftest TAP suite.
Problem with the old approach
------------------------------
On every push both workflows independently rebuilt all images from scratch.
A typical PR triggered six separate image builds for PG14-17 (two
workflows × four versions), each re-compiling Postgres + Citus + pg_auto_failover
from source. The base→citus→build chain alone takes ~8-10 min per version.
New ci.yml (unified, single workflow)
--------------------------------------
One build matrix job per Postgres major version replaces both workflows'
build stages:
style ──► build (PG14..17) ──► pytest (PG14-17)
├─► pgaftest (PG14-17, all schedules)
├─► installcheck (PG14-17)
└─► upgrade (PG16)
Each build job:
- Uses Docker Buildx with GHA layer cache (scope build-pgNN) so the
expensive base→citus→build chain is a cache hit on subsequent runs
when neither the Dockerfile nor Postgres/Citus versions have changed.
- Builds all required targets sequentially (test, run, debian, testrun,
pgaftest for PG17); Docker's local daemon cache means the shared build
layers are compiled once and reused for every subsequent target in the
same job.
- Uploads each image as a GitHub Actions artifact (retention: 1 day) for
downstream test jobs to download and load.
The pytest suite and pgaftest suite both need: [build], not each other,
so all test jobs run fully in parallel after the build fan-out.
New base-images.yml
--------------------
Separate workflow maintains cached base images in GHCR for cross-run
layer reuse. Two schedules:
nightly (02:00 UTC) — rebuilds PG19 from PostgreSQL master so its
frequently-changing headers stay current.
weekly (Mon 03:00) — checks github.com/postgres/postgres for new
minor-release tags (REL_NN_X pattern, numeric
minor only). If a new tag appears, rebuilds
the base image, pushes to
ghcr.io/hapostgres/pg-cache:pgNN, and commits
the updated .github/pg-base-versions/pgNN.ref
so the next weekly check knows what's already
been built.
manual — workflow_dispatch with an optional pgversion
input to trigger a rebuild for a specific
major version immediately.
.github/pg-base-versions/
--------------------------
Seed files recording the current latest minor release for each stable
major version (PG14-18), used by the weekly check job as its baseline.
dimitri
added
Developer productivity
matrix.* is only resolved within steps, not at the job-level if condition. Move the manual pgversion filter into the detect step instead, using an early exit so all downstream steps (gated on rebuild == 'true') are skipped naturally. Also drop the now-redundant PGVERSION_STR matrix field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Both
run-tests.ymlandrun-pgaftest.ymlindependently built all images from scratch on every push. A typical PR triggered six separate image builds for PG14–17 (two workflows × four versions), each re-compiling Postgres + Citus + pg_auto_failover from source. Thebase→citus→buildchain alone takes ~8–10 min per version.Changes
ci.yml— single unified workflowReplaces both old workflows. One
buildmatrix job per Postgres major version:Each build job:
build-pgNN) — the expensivebase→citus→buildchain is a cache hit on subsequent runs when neither the Dockerfile nor Postgres/Citus versions changetest,run,debian,testruntargets in the same jobThe style checker also runs once instead of twice.
base-images.yml— maintains long-lived base image cache in GHCRghcr.io/hapostgres/pg-cache:pgNN— theci.ymlbuild jobs pull this as--cache-from.Three triggers:
master(frequently-changing headers)github.com/postgres/postgrestags for new stable minor releases (REL_NN_Xpattern); rebuilds + pushes cache + commits updated.github/pg-base-versions/pgNN.refonly when a new tag appearsworkflow_dispatch— rebuild any single version on demand.github/pg-base-versions/Seed files recording the current latest minor release for each stable major (PG14–18). The weekly check job uses these as its baseline to detect new releases.