From 01fa9f98ee467d22d32c1b4cd2188c6868988dcf Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sun, 12 Jul 2026 16:20:38 +0200 Subject: [PATCH 1/2] ci: unified workflow with build-once image sharing and cross-run layer cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the two independent workflows (run-tests.yml, run-pgaftest.yml) with a single ci.yml that builds each Postgres version's images once and shares them across both the pytest suite and the pgaftest TAP suite. Problem with the old approach ------------------------------ On every push both workflows independently rebuilt all images from scratch. A typical PR triggered six separate image builds for PG14-17 (two workflows × four versions), each re-compiling Postgres + Citus + pg_auto_failover from source. The base→citus→build chain alone takes ~8-10 min per version. New ci.yml (unified, single workflow) -------------------------------------- One build matrix job per Postgres major version replaces both workflows' build stages: style ──► build (PG14..17) ──► pytest (PG14-17) ├─► pgaftest (PG14-17, all schedules) ├─► installcheck (PG14-17) └─► upgrade (PG16) Each build job: - Uses Docker Buildx with GHA layer cache (scope build-pgNN) so the expensive base→citus→build chain is a cache hit on subsequent runs when neither the Dockerfile nor Postgres/Citus versions have changed. - Builds all required targets sequentially (test, run, debian, testrun, pgaftest for PG17); Docker's local daemon cache means the shared build layers are compiled once and reused for every subsequent target in the same job. - Uploads each image as a GitHub Actions artifact (retention: 1 day) for downstream test jobs to download and load. The pytest suite and pgaftest suite both need: [build], not each other, so all test jobs run fully in parallel after the build fan-out. New base-images.yml -------------------- Separate workflow maintains cached base images in GHCR for cross-run layer reuse. Two schedules: nightly (02:00 UTC) — rebuilds PG19 from PostgreSQL master so its frequently-changing headers stay current. weekly (Mon 03:00) — checks github.com/postgres/postgres for new minor-release tags (REL_NN_X pattern, numeric minor only). If a new tag appears, rebuilds the base image, pushes to ghcr.io/hapostgres/pg-cache:pgNN, and commits the updated .github/pg-base-versions/pgNN.ref so the next weekly check knows what's already been built. manual — workflow_dispatch with an optional pgversion input to trigger a rebuild for a specific major version immediately. .github/pg-base-versions/ -------------------------- Seed files recording the current latest minor release for each stable major version (PG14-18), used by the weekly check job as its baseline. --- .github/pg-base-versions/pg14.ref | 1 + .github/pg-base-versions/pg15.ref | 1 + .github/pg-base-versions/pg16.ref | 1 + .github/pg-base-versions/pg17.ref | 1 + .github/pg-base-versions/pg18.ref | 1 + .github/workflows/base-images.yml | 178 ++++++++++++ .github/workflows/ci.yml | 432 +++++++++++++++++++++++++++++ .github/workflows/run-pgaftest.yml | 358 ------------------------ .github/workflows/run-tests.yml | 101 ------- 9 files changed, 615 insertions(+), 459 deletions(-) create mode 100644 .github/pg-base-versions/pg14.ref create mode 100644 .github/pg-base-versions/pg15.ref create mode 100644 .github/pg-base-versions/pg16.ref create mode 100644 .github/pg-base-versions/pg17.ref create mode 100644 .github/pg-base-versions/pg18.ref create mode 100644 .github/workflows/base-images.yml create mode 100644 .github/workflows/ci.yml delete mode 100644 .github/workflows/run-pgaftest.yml delete mode 100644 .github/workflows/run-tests.yml diff --git a/.github/pg-base-versions/pg14.ref b/.github/pg-base-versions/pg14.ref new file mode 100644 index 000000000..9444993bb --- /dev/null +++ b/.github/pg-base-versions/pg14.ref @@ -0,0 +1 @@ +REL_14_23 diff --git a/.github/pg-base-versions/pg15.ref b/.github/pg-base-versions/pg15.ref new file mode 100644 index 000000000..4b29bd679 --- /dev/null +++ b/.github/pg-base-versions/pg15.ref @@ -0,0 +1 @@ +REL_15_18 diff --git a/.github/pg-base-versions/pg16.ref b/.github/pg-base-versions/pg16.ref new file mode 100644 index 000000000..2b994967e --- /dev/null +++ b/.github/pg-base-versions/pg16.ref @@ -0,0 +1 @@ +REL_16_14 diff --git a/.github/pg-base-versions/pg17.ref b/.github/pg-base-versions/pg17.ref new file mode 100644 index 000000000..56f93698c --- /dev/null +++ b/.github/pg-base-versions/pg17.ref @@ -0,0 +1 @@ +REL_17_10 diff --git a/.github/pg-base-versions/pg18.ref b/.github/pg-base-versions/pg18.ref new file mode 100644 index 000000000..71e4c108a --- /dev/null +++ b/.github/pg-base-versions/pg18.ref @@ -0,0 +1 @@ +REL_18_4 diff --git a/.github/workflows/base-images.yml b/.github/workflows/base-images.yml new file mode 100644 index 000000000..edb81fd91 --- /dev/null +++ b/.github/workflows/base-images.yml @@ -0,0 +1,178 @@ +name: Base images + +# Maintains cached base images in GHCR so CI build jobs skip re-compiling +# Postgres and Citus from source on every PR. +# +# Two triggers: +# +# nightly — rebuilds PG19 (tracking PostgreSQL master), every day at 02:00 +# UTC. PG19 is in active development so its headers change often +# enough that a daily refresh is worthwhile. +# +# weekly — checks whether a new Postgres minor release has been tagged on +# github.com/postgres/postgres for each stable major version +# (PG14–18). If so, rebuilds and pushes a fresh base image and +# commits the updated .github/pg-base-versions/pgNN.ref file so +# the next weekly check knows what was already built. +# +# manual — workflow_dispatch lets you rebuild any single major version on +# demand (e.g. right after a minor release ships, without waiting +# for the next weekly cron). +# +# Image naming in GHCR: +# ghcr.io/hapostgres/pg-cache:pgNN +# +# The ci.yml build jobs pull this as --cache-from, which makes the +# base→citus→build chain a cache hit when Postgres/Citus haven't changed. + +on: + schedule: + # Nightly at 02:00 UTC — rebuilds PG19 (master). + - cron: '0 2 * * *' + # Weekly on Monday 03:00 UTC — checks stable minor versions PG14-18. + - cron: '0 3 * * 1' + + workflow_dispatch: + inputs: + pgversion: + description: 'Postgres major version to rebuild (e.g. 17). Leave blank to run the full weekly check.' + required: false + type: string + +env: + REGISTRY: ghcr.io + CACHE_IMAGE: ghcr.io/${{ github.repository_owner }}/pg-cache + +jobs: + # --------------------------------------------------------------------------- + # Nightly: rebuild PG19 from PostgreSQL master. + # --------------------------------------------------------------------------- + nightly-pg19: + name: Nightly rebuild (PG19 / master) + if: > + github.event_name == 'schedule' && github.event.schedule == '0 2 * * *' || + github.event_name == 'workflow_dispatch' && github.event.inputs.pgversion == '19' + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to GHCR + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Generate git-version.h + run: make version + + - name: Build and cache PG19 base image + run: | + docker buildx build \ + --cache-from type=registry,ref=${{ env.CACHE_IMAGE }}:pg19 \ + --cache-to type=registry,ref=${{ env.CACHE_IMAGE }}:pg19,mode=max \ + --build-arg PGVERSION=19 \ + --build-arg CITUSTAG=none \ + --target build \ + . + + # --------------------------------------------------------------------------- + # Weekly: detect new Postgres minor releases for stable versions (PG14-18). + # --------------------------------------------------------------------------- + check-minor-releases: + name: Check minor releases (PG${{ matrix.PGVERSION }}) + if: > + github.event_name == 'schedule' && github.event.schedule == '0 3 * * 1' || + github.event_name == 'workflow_dispatch' && ( + github.event.inputs.pgversion == '' || + github.event.inputs.pgversion == matrix.PGVERSION_STR + ) + runs-on: ubuntu-latest + permissions: + contents: write + packages: write + strategy: + fail-fast: false + matrix: + include: + - { PGVERSION: 14, PGVERSION_STR: '14', CITUSTAG: v12.1.5 } + - { PGVERSION: 15, PGVERSION_STR: '15', CITUSTAG: v12.1.5 } + - { PGVERSION: 16, PGVERSION_STR: '16', CITUSTAG: v13.2.0 } + - { PGVERSION: 17, PGVERSION_STR: '17', CITUSTAG: v13.2.0 } + - { PGVERSION: 18, PGVERSION_STR: '18', CITUSTAG: v14.1.0 } + + steps: + - uses: actions/checkout@v4 + with: + # Need push access to update ref files. + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Detect latest Postgres ${{ matrix.PGVERSION }} minor release + id: detect + run: | + LATEST=$(git ls-remote --tags https://github.com/postgres/postgres \ + "refs/tags/REL_${{ matrix.PGVERSION }}_*" \ + | grep -v '\^{}' \ + | awk '{print $2}' \ + | sed 's|refs/tags/||' \ + | grep -E "^REL_${{ matrix.PGVERSION }}_[0-9]+$" \ + | sort -V \ + | tail -1) + + STORED=$(cat .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref 2>/dev/null || echo "none") + + echo "latest=${LATEST}" >> $GITHUB_OUTPUT + echo "stored=${STORED}" >> $GITHUB_OUTPUT + + if [ "${LATEST}" != "${STORED}" ] && [ -n "${LATEST}" ]; then + echo "rebuild=true" >> $GITHUB_OUTPUT + echo "PG${{ matrix.PGVERSION }}: new minor release ${LATEST} (was ${STORED}), rebuilding" + else + echo "rebuild=false" >> $GITHUB_OUTPUT + echo "PG${{ matrix.PGVERSION }}: already at ${STORED}, nothing to do" + fi + + - name: Set up Docker Buildx + if: steps.detect.outputs.rebuild == 'true' + uses: docker/setup-buildx-action@v3 + + - name: Log in to GHCR + if: steps.detect.outputs.rebuild == 'true' + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Generate git-version.h + if: steps.detect.outputs.rebuild == 'true' + run: make version + + - name: Build and cache PG${{ matrix.PGVERSION }} base image + if: steps.detect.outputs.rebuild == 'true' + run: | + docker buildx build \ + --cache-from type=registry,ref=${{ env.CACHE_IMAGE }}:pg${{ matrix.PGVERSION }} \ + --cache-to type=registry,ref=${{ env.CACHE_IMAGE }}:pg${{ matrix.PGVERSION }},mode=max \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ matrix.CITUSTAG }} \ + --target build \ + . + + - name: Update stored ref + if: steps.detect.outputs.rebuild == 'true' + run: | + mkdir -p .github/pg-base-versions + echo "${{ steps.detect.outputs.latest }}" > .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git add .github/pg-base-versions/pg${{ matrix.PGVERSION }}.ref + git commit -m "ci: update PG${{ matrix.PGVERSION }} base image to ${{ steps.detect.outputs.latest }}" + git push diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 000000000..d1182b399 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,432 @@ +name: CI + +# Unified workflow: build images once, share across both test suites. +# +# Job graph per push/PR: +# +# style ──► build (PG14) ──► pytest (PG14, multi/single/monitor/ssl/citus/tablespaces) +# │ ├─► pgaftest (PG14, quick/node/ssl) +# │ └─► installcheck (PG14) +# ├─► build (PG15) ──► ... +# ├─► build (PG16) ──► ... +# └─► build (PG17) ──► pytest (PG17, all schedules incl. slow ones) +# ├─► pgaftest (PG17, all schedules incl. slow ones) +# ├─► installcheck (PG17) +# └─► upgrade +# +# Each build job: +# 1. Pulls GHA layer cache (scope build-pgNN) to skip re-compiling Postgres+Citus +# on unchanged code. Writes the cache back so subsequent runs stay warm. +# 2. Builds all targets (test, run, debian, testrun; pgaftest for PG17 only) +# sequentially — Docker's local layer cache means the base→citus→build chain +# is compiled once and reused for every subsequent target in the same job. +# 3. Uploads each image as a GitHub Actions artifact (retention: 1 day) for +# downstream test jobs to download. + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + # --------------------------------------------------------------------------- + # Style gate — runs once, not once per PG version. + # --------------------------------------------------------------------------- + style: + name: Style check + runs-on: ubuntu-latest + container: citus/stylechecker:no-py + steps: + - uses: actions/checkout@v4 + + - name: Set safe directory for git + run: git config --global --add safe.directory ${GITHUB_WORKSPACE} + + - name: Check C formatting + run: citus_indent --check + + - name: Check banned functions + run: ci/banned.h.sh + + # --------------------------------------------------------------------------- + # Build all images for one Postgres major version. + # Uploads everything as artifacts; downstream jobs download only what they need. + # --------------------------------------------------------------------------- + build: + name: Build images (PG${{ matrix.PGVERSION }}) + needs: style + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + PGVERSION: [14, 15, 16, 17] + + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Set CITUSTAG for PG${{ matrix.PGVERSION }} + run: | + case "${{ matrix.PGVERSION }}" in + 14|15) echo "CITUSTAG=v12.1.5" >> $GITHUB_ENV ;; + *) echo "CITUSTAG=v13.2.0" >> $GITHUB_ENV ;; + esac + + - name: Generate git-version.h + run: make version + + # Build the fat build image (base → citus → build → test). + # This is the most expensive step; GHA layer cache avoids re-compiling + # Postgres+Citus when the Dockerfile and source are unchanged. + - name: Build test image (pytest suite) + run: | + docker buildx build \ + --cache-from type=gha,scope=build-pg${{ matrix.PGVERSION }} \ + --cache-to type=gha,scope=build-pg${{ matrix.PGVERSION }},mode=max \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ env.CITUSTAG }} \ + --target test \ + -t pg_auto_failover_test:pg${{ matrix.PGVERSION }} \ + --load \ + . + + # Slim runtime image. Reuses build-stage layers from the local daemon + # cache populated by the test build above — no re-compilation. + - name: Build run image (pgaftest cluster node) + run: | + docker buildx build \ + --cache-from type=gha,scope=build-pg${{ matrix.PGVERSION }} \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ env.CITUSTAG }} \ + --target run \ + -t pgaf:run-pg${{ matrix.PGVERSION }} \ + --load \ + . + + # debian target extends run (pg_createcluster layer) — fast. + - name: Build debian image (node schedule) + run: | + docker buildx build \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ env.CITUSTAG }} \ + --target debian \ + -t pgaf:debian-pg${{ matrix.PGVERSION }} \ + --load \ + . + + # testrun target extends run (adds server-dev headers) — fast. + - name: Build testrun image (installcheck) + run: | + docker buildx build \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ env.CITUSTAG }} \ + --target testrun \ + -t pgaf:testrun-pg${{ matrix.PGVERSION }} \ + --load \ + . + + # pgaftest runner container: only one version needed (it runs pgaftest + # itself and drives cluster nodes via Docker-out-of-Docker). + - name: Build pgaftest runner image + if: matrix.PGVERSION == 17 + run: | + docker buildx build \ + --cache-from type=gha,scope=pgaftest \ + --cache-to type=gha,scope=pgaftest,mode=max \ + --build-arg PGVERSION=${{ matrix.PGVERSION }} \ + --build-arg CITUSTAG=${{ env.CITUSTAG }} \ + --target pgaftest \ + -t pgaf:pgaftest \ + --load \ + . + + - name: Save images to archives + run: | + # test image: use zstd (matches make save-test-image / load-test-image) + docker save pg_auto_failover_test:pg${{ matrix.PGVERSION }} \ + | zstd -T0 -3 > /tmp/pg_auto_failover_test-pg${{ matrix.PGVERSION }}.tar.zst + + docker save pgaf:run-pg${{ matrix.PGVERSION }} \ + | gzip > /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz + + docker save pgaf:debian-pg${{ matrix.PGVERSION }} \ + | gzip > /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz + + docker save pgaf:testrun-pg${{ matrix.PGVERSION }} \ + | gzip > /tmp/pgaf-testrun-pg${{ matrix.PGVERSION }}.tar.gz + + if [ "${{ matrix.PGVERSION }}" = "17" ]; then + docker save pgaf:pgaftest | gzip > /tmp/pgaf-pgaftest.tar.gz + fi + + - name: Upload test image + uses: actions/upload-artifact@v4 + with: + name: pg-test-image-${{ matrix.PGVERSION }} + path: /tmp/pg_auto_failover_test-pg${{ matrix.PGVERSION }}.tar.zst + retention-days: 1 + + - name: Upload run image + uses: actions/upload-artifact@v4 + with: + name: pgaf-run-image-pg${{ matrix.PGVERSION }} + path: /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz + retention-days: 1 + + - name: Upload debian image + uses: actions/upload-artifact@v4 + with: + name: pgaf-debian-image-pg${{ matrix.PGVERSION }} + path: /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz + retention-days: 1 + + - name: Upload testrun image + uses: actions/upload-artifact@v4 + with: + name: pgaf-testrun-image-pg${{ matrix.PGVERSION }} + path: /tmp/pgaf-testrun-pg${{ matrix.PGVERSION }}.tar.gz + retention-days: 1 + + - name: Upload pgaftest runner image + if: matrix.PGVERSION == 17 + uses: actions/upload-artifact@v4 + with: + name: pgaftest-image + path: /tmp/pgaf-pgaftest.tar.gz + retention-days: 1 + + # --------------------------------------------------------------------------- + # pytest suite (Python integration tests, tests/ directory). + # --------------------------------------------------------------------------- + pytest: + name: Run test (${{ matrix.PGVERSION }}, ${{ matrix.TEST }}) + needs: build + runs-on: ubuntu-latest + timeout-minutes: 15 + strategy: + fail-fast: false + matrix: + PGVERSION: [14, 15, 16, 17] + TEST: [multi, single, monitor, ssl, citus] + include: + - PGVERSION: 14 + TEST: tablespaces + + steps: + - uses: actions/checkout@v4 + + - name: Download test image (PG${{ matrix.PGVERSION }}) + uses: actions/download-artifact@v4 + with: + name: pg-test-image-${{ matrix.PGVERSION }} + + - name: Load test image + run: | + PGVERSION=${{ matrix.PGVERSION }} make load-test-image + + - name: Run test + env: + PGVERSION: ${{ matrix.PGVERSION }} + TEST: ${{ matrix.TEST }} + TRAVIS_BUILD_DIR: ${{ github.workspace }} + run: make run-test-prebuilt + + # --------------------------------------------------------------------------- + # pgaftest TAP suite. + # Fast schedules (quick, node, ssl) run on all four PG versions. + # Slow schedules (multi-*, citus-*) are PG17-only to keep total job count low. + # --------------------------------------------------------------------------- + pgaftest: + name: pgaftest / ${{ matrix.schedule }} (PG${{ matrix.PGVERSION }}) + needs: build + runs-on: ubuntu-latest + timeout-minutes: 40 + strategy: + fail-fast: false + matrix: + include: + - { PGVERSION: 14, schedule: quick } + - { PGVERSION: 15, schedule: quick } + - { PGVERSION: 16, schedule: quick } + - { PGVERSION: 17, schedule: quick } + - { PGVERSION: 14, schedule: node } + - { PGVERSION: 15, schedule: node } + - { PGVERSION: 16, schedule: node } + - { PGVERSION: 17, schedule: node } + - { PGVERSION: 14, schedule: ssl } + - { PGVERSION: 15, schedule: ssl } + - { PGVERSION: 16, schedule: ssl } + - { PGVERSION: 17, schedule: ssl } + - { PGVERSION: 17, schedule: multi-alternate } + - { PGVERSION: 17, schedule: multi-misc } + - { PGVERSION: 17, schedule: multi-async } + - { PGVERSION: 17, schedule: citus-1 } + - { PGVERSION: 17, schedule: citus-2 } + + steps: + - uses: actions/checkout@v4 + + - name: Download pgaf:run image (PG${{ matrix.PGVERSION }}) + uses: actions/download-artifact@v4 + with: + name: pgaf-run-image-pg${{ matrix.PGVERSION }} + path: /tmp + + - name: Load pgaf:run image + run: | + docker load < /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz + docker tag pgaf:run-pg${{ matrix.PGVERSION }} pgaf:run + + - name: Download pgaf:debian image (PG${{ matrix.PGVERSION }}) + if: matrix.schedule == 'node' + uses: actions/download-artifact@v4 + with: + name: pgaf-debian-image-pg${{ matrix.PGVERSION }} + path: /tmp + + - name: Load pgaf:debian image + if: matrix.schedule == 'node' + run: | + docker load < /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz + docker tag pgaf:debian-pg${{ matrix.PGVERSION }} pgaf:debian + + - name: Download pgaf:pgaftest runner image + uses: actions/download-artifact@v4 + with: + name: pgaftest-image + path: /tmp + + - name: Load pgaf:pgaftest image + run: docker load < /tmp/pgaf-pgaftest.tar.gz + + - name: Run schedule ${{ matrix.schedule }} (PG${{ matrix.PGVERSION }}) + timeout-minutes: 35 + run: | + mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest + DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) + docker run --rm \ + --user root \ + --group-add "${DOCKER_GID}" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp/pgaftest:/tmp/pgaftest \ + -v "$(pwd)":/work:ro \ + -w /work \ + -e PGAF_IMAGE=pgaf:run \ + -e PGVERSION=${{ matrix.PGVERSION }} \ + -e PGAF_DEBIAN_IMAGE=${{ matrix.schedule == 'node' && 'pgaf:debian' || '' }} \ + -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ + pgaf:pgaftest \ + pgaftest run --schedule tests/tap/schedules/${{ matrix.schedule }}.sch + + # --------------------------------------------------------------------------- + # SQL regression suite (pg_regress installcheck). + # --------------------------------------------------------------------------- + installcheck: + name: pgaftest / installcheck (PG${{ matrix.PGVERSION }}) + needs: build + runs-on: ubuntu-latest + timeout-minutes: 20 + strategy: + fail-fast: false + matrix: + PGVERSION: [14, 15, 16, 17] + + steps: + - uses: actions/checkout@v4 + + - name: Download pgaf:testrun image (PG${{ matrix.PGVERSION }}) + uses: actions/download-artifact@v4 + with: + name: pgaf-testrun-image-pg${{ matrix.PGVERSION }} + path: /tmp + + - name: Load pgaf:testrun image + run: docker load < /tmp/pgaf-testrun-pg${{ matrix.PGVERSION }}.tar.gz + + - name: Download pgaf:pgaftest runner image + uses: actions/download-artifact@v4 + with: + name: pgaftest-image + path: /tmp + + - name: Load pgaf:pgaftest image + run: docker load < /tmp/pgaf-pgaftest.tar.gz + + - name: Generate git-version.h + run: make version + + - name: Run installcheck + timeout-minutes: 15 + run: | + mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest + DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) + docker run --rm \ + --user root \ + --group-add "${DOCKER_GID}" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp/pgaftest:/tmp/pgaftest \ + -v "$(pwd)":/work:ro \ + -w /work \ + -e PGVERSION=${{ matrix.PGVERSION }} \ + -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ + pgaf:pgaftest \ + pgaftest run tests/tap/specs/installcheck.pgaf + + # --------------------------------------------------------------------------- + # Upgrade test (PG16, binary + extension swap without container restart). + # --------------------------------------------------------------------------- + upgrade: + name: pgaftest / upgrade + needs: build + runs-on: ubuntu-latest + timeout-minutes: 30 + env: + PGVERSION: 16 + + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Download pgaf:pgaftest runner image + uses: actions/download-artifact@v4 + with: + name: pgaftest-image + path: /tmp + + - name: Load pgaf:pgaftest image + run: docker load < /tmp/pgaf-pgaftest.tar.gz + + - name: Generate git-version.h + run: make version + + - name: Build pgaf:next (current branch, PG${{ env.PGVERSION }}) + run: make -C tests/upgrade pgaf-next PGVERSION=${{ env.PGVERSION }} + + - name: Build pgaf:current (previous release, PG${{ env.PGVERSION }}) + run: make -C tests/upgrade pgaf-current PGVERSION=${{ env.PGVERSION }} + + - name: Run upgrade spec + timeout-minutes: 25 + run: | + mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest + DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) + docker run --rm \ + --user root \ + --group-add "${DOCKER_GID}" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /tmp/pgaftest:/tmp/pgaftest \ + -v "$(pwd)":/work:ro \ + -w /work \ + -e PGVERSION=${{ env.PGVERSION }} \ + -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ + pgaf:pgaftest \ + pgaftest run tests/tap/specs/upgrade.pgaf diff --git a/.github/workflows/run-pgaftest.yml b/.github/workflows/run-pgaftest.yml deleted file mode 100644 index b6a9085c8..000000000 --- a/.github/workflows/run-pgaftest.yml +++ /dev/null @@ -1,358 +0,0 @@ -name: pgaftest - -on: - push: - branches: - - main - pull_request: - branches: - - main - workflow_dispatch: - -# One run per branch; cancel in-progress runs on new push. -concurrency: - group: pgaftest-${{ github.ref }} - cancel-in-progress: true - -jobs: - style_checker: - name: Style check - runs-on: ubuntu-latest - container: citus/stylechecker:no-py - steps: - - name: Checkout repository - uses: actions/checkout@v7.0.0 - - - name: Set safe directory for git - run: git config --global --add safe.directory ${GITHUB_WORKSPACE} - - - name: Check C formatting - run: citus_indent --check - - - name: Check banned functions - run: ci/banned.h.sh - - # --------------------------------------------------------------------------- - # Build one pgaf:run image per Postgres version and upload each as an - # artifact. All downstream test jobs download only the version they need. - # --------------------------------------------------------------------------- - build-images: - name: Build image (PG${{ matrix.PGVERSION }}) - needs: style_checker - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - PGVERSION: [14, 15, 16, 17] - - steps: - - uses: actions/checkout@v7.0.0 - - - name: Generate git-version.h - run: make version - - - name: Build pgaf:run image (PG${{ matrix.PGVERSION }}) - run: | - # Match the CITUSTAG-per-PGVERSION logic from the top-level Makefile: - # PG14–15 require Citus v12 (last version supporting those releases). - # PG16+ use the current default. - case "${{ matrix.PGVERSION }}" in - 14|15) CITUSTAG=v12.1.5 ;; - *) CITUSTAG=v13.2.0 ;; - esac - docker build \ - --build-arg PGVERSION=${{ matrix.PGVERSION }} \ - --build-arg CITUSTAG=${CITUSTAG} \ - --target run \ - -t pgaf:run-pg${{ matrix.PGVERSION }} \ - . - - - name: Build pgaf:debian image (PG${{ matrix.PGVERSION }}) - run: | - # Layer cache from the run build above covers the run stage; - # this step only adds the pg_createcluster layer on top. - case "${{ matrix.PGVERSION }}" in - 14|15) CITUSTAG=v12.1.5 ;; - *) CITUSTAG=v13.2.0 ;; - esac - docker build \ - --build-arg PGVERSION=${{ matrix.PGVERSION }} \ - --build-arg CITUSTAG=${CITUSTAG} \ - --target debian \ - -t pgaf:debian-pg${{ matrix.PGVERSION }} \ - . - - - name: Save images to tarballs - run: | - docker save pgaf:run-pg${{ matrix.PGVERSION }} \ - | gzip > /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz - docker save pgaf:debian-pg${{ matrix.PGVERSION }} \ - | gzip > /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz - - - name: Upload run image artifact - uses: actions/upload-artifact@v7.0.1 - with: - name: pgaf-run-image-pg${{ matrix.PGVERSION }} - path: /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz - retention-days: 1 - - - name: Upload debian image artifact - uses: actions/upload-artifact@v7.0.1 - with: - name: pgaf-debian-image-pg${{ matrix.PGVERSION }} - path: /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz - retention-days: 1 - - # --------------------------------------------------------------------------- - # Build the pgaf:pgaftest image once from the PG17 build image. - # The Dockerfile pgaftest target bundles the pgaftest binary together with - # its runtime dependencies (libpq5 from PGDG, docker-ce-cli, and the - # docker-compose-plugin) so that test jobs run pgaftest inside this - # container via Docker-out-of-Docker rather than installing libpq on the - # bare runner. That eliminates the fragile apt-get dependency on the - # pre-installed Microsoft package repositories on GitHub's runner image. - # --------------------------------------------------------------------------- - build-pgaftest: - name: Build pgaftest image - needs: style_checker - runs-on: ubuntu-latest - env: - PGVERSION: 17 - - steps: - - uses: actions/checkout@v7.0.0 - - - name: Generate git-version.h - run: make version - - - name: Build pgaf:pgaftest image (PG${{ env.PGVERSION }}) - run: | - docker build \ - --build-arg PGVERSION=${{ env.PGVERSION }} \ - --build-arg CITUSTAG=v13.2.0 \ - --target pgaftest \ - -t pgaf:pgaftest \ - . - - - name: Save image to tarball - run: docker save pgaf:pgaftest | gzip > /tmp/pgaf-pgaftest.tar.gz - - - name: Upload pgaftest image artifact - uses: actions/upload-artifact@v7.0.1 - with: - name: pgaftest-image - path: /tmp/pgaf-pgaftest.tar.gz - retention-days: 1 - - # --------------------------------------------------------------------------- - # Run test schedules. Fast schedules (quick, node, ssl) run against all four - # Postgres versions. Slow schedules (multi-*, citus-*) run PG17 only. - # Total: 3×4 + 5×1 = 17 test jobs — keeps GitHub shared-runner queue short. - # --------------------------------------------------------------------------- - test: - name: pgaftest / ${{ matrix.schedule }} (PG${{ matrix.PGVERSION }}) - needs: [build-images, build-pgaftest] - runs-on: ubuntu-latest - timeout-minutes: 40 - strategy: - fail-fast: false - matrix: - # Fast/medium schedules run against all four Postgres versions. - # Slow schedules (multi-*, citus-*) run PG17 only: they exercise - # pg_auto_failover FSM logic, not Postgres version-specific code paths, - # and restricting them reduces total job count from 36 to 17 — avoiding - # GitHub Actions shared-runner queue saturation that was adding 10-15 min - # of queue wait to every run. - include: - # quick: basic_operation, basic_operation_listen_flag, config_get_set, skip_pg_hba - - { PGVERSION: 14, schedule: quick } - - { PGVERSION: 15, schedule: quick } - - { PGVERSION: 16, schedule: quick } - - { PGVERSION: 17, schedule: quick } - # node: create_standby_with_pgdata, maintenance_and_drop, auth, - # monitor_disabled, replace_monitor, extension_update, - # debian_clusters, tablespaces - - { PGVERSION: 14, schedule: node } - - { PGVERSION: 15, schedule: node } - - { PGVERSION: 16, schedule: node } - - { PGVERSION: 17, schedule: node } - # ssl: enable_ssl, ssl_self_signed, ssl_cert - - { PGVERSION: 14, schedule: ssl } - - { PGVERSION: 15, schedule: ssl } - - { PGVERSION: 16, schedule: ssl } - - { PGVERSION: 17, schedule: ssl } - # slow schedules: PG17 only - - { PGVERSION: 17, schedule: multi-alternate } # multi_alternate - - { PGVERSION: 17, schedule: multi-misc } # multi_standbys, multi_maintenance, ensure, multi_ifdown - - { PGVERSION: 17, schedule: multi-async } # multi_async - - { PGVERSION: 17, schedule: citus-1 } # citus_cluster_name, citus_force_failover, citus_multi_standbys - - { PGVERSION: 17, schedule: citus-2 } # basic_citus_operation, nonha_citus_operation, citus_skip_pg_hba - - steps: - - uses: actions/checkout@v7.0.0 - - - name: Download pgaf:run image (PG${{ matrix.PGVERSION }}) - uses: actions/download-artifact@v8.0.1 - with: - name: pgaf-run-image-pg${{ matrix.PGVERSION }} - path: /tmp - - - name: Load pgaf:run image into Docker - run: | - docker load < /tmp/pgaf-run-pg${{ matrix.PGVERSION }}.tar.gz - # Alias to the name pgaftest expects via PGAF_IMAGE - docker tag pgaf:run-pg${{ matrix.PGVERSION }} pgaf:run - - - name: Download pgaf:debian image (PG${{ matrix.PGVERSION }}) - if: matrix.schedule == 'node' - uses: actions/download-artifact@v8.0.1 - with: - name: pgaf-debian-image-pg${{ matrix.PGVERSION }} - path: /tmp - - - name: Load pgaf:debian image into Docker - if: matrix.schedule == 'node' - run: | - docker load < /tmp/pgaf-debian-pg${{ matrix.PGVERSION }}.tar.gz - docker tag pgaf:debian-pg${{ matrix.PGVERSION }} pgaf:debian - - - name: Download pgaf:pgaftest image - uses: actions/download-artifact@v8.0.1 - with: - name: pgaftest-image - path: /tmp - - - name: Load pgaf:pgaftest image into Docker - run: docker load < /tmp/pgaf-pgaftest.tar.gz - - - name: Run schedule ${{ matrix.schedule }} - timeout-minutes: 35 - run: | - mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest - DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) - docker run --rm \ - --user root \ - --group-add "${DOCKER_GID}" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /tmp/pgaftest:/tmp/pgaftest \ - -v "$(pwd)":/work:ro \ - -w /work \ - -e PGAF_IMAGE=pgaf:run \ - -e PGVERSION=${{ matrix.PGVERSION }} \ - -e PGAF_DEBIAN_IMAGE=${{ matrix.schedule == 'node' && 'pgaf:debian' || '' }} \ - -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ - pgaf:pgaftest \ - pgaftest run --schedule tests/tap/schedules/${{ matrix.schedule }}.sch - - # --------------------------------------------------------------------------- - # installcheck: SQL regression suite via pg_regress. - # Runs on all supported Postgres versions; builds pgaf:testrun inline since - # it needs postgresql-server-dev which isn't in the run image. - # --------------------------------------------------------------------------- - installcheck: - name: pgaftest / installcheck (PG${{ matrix.PGVERSION }}) - needs: build-pgaftest - runs-on: ubuntu-latest - timeout-minutes: 20 - strategy: - fail-fast: false - matrix: - PGVERSION: [14, 15, 16, 17] - - steps: - - uses: actions/checkout@v7.0.0 - - - name: Download pgaf:pgaftest image - uses: actions/download-artifact@v8.0.1 - with: - name: pgaftest-image - path: /tmp - - - name: Load pgaf:pgaftest image into Docker - run: docker load < /tmp/pgaf-pgaftest.tar.gz - - - name: Generate git-version.h - run: make version - - - name: Build pgaf:testrun image (PG${{ matrix.PGVERSION }}) - run: | - case "${{ matrix.PGVERSION }}" in - 14|15) CITUSTAG=v12.1.5 ;; - *) CITUSTAG=v13.2.0 ;; - esac - docker build \ - --build-arg PGVERSION=${{ matrix.PGVERSION }} \ - --build-arg CITUSTAG=${CITUSTAG} \ - --target testrun \ - -t pgaf:testrun \ - . - - - name: Run installcheck spec - timeout-minutes: 15 - run: | - mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest - DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) - docker run --rm \ - --user root \ - --group-add "${DOCKER_GID}" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /tmp/pgaftest:/tmp/pgaftest \ - -v "$(pwd)":/work:ro \ - -w /work \ - -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ - pgaf:pgaftest \ - pgaftest run tests/tap/specs/installcheck.pgaf - - # --------------------------------------------------------------------------- - # Upgrade test: binary + extension swap without container restart. - # Pinned to PG16 (the upgrade path always runs N-1 → current). - # Builds its own pgaf:next and pgaf:current images from the upgrade Makefile. - # --------------------------------------------------------------------------- - upgrade: - name: pgaftest / upgrade - needs: build-pgaftest - runs-on: ubuntu-latest - timeout-minutes: 30 - env: - PGVERSION: 16 - - steps: - - uses: actions/checkout@v7.0.0 - with: - # Full history needed so PREV_TAG auto-detection finds the right tag. - fetch-depth: 0 - - - name: Download pgaf:pgaftest image - uses: actions/download-artifact@v8.0.1 - with: - name: pgaftest-image - path: /tmp - - - name: Load pgaf:pgaftest image into Docker - run: docker load < /tmp/pgaf-pgaftest.tar.gz - - - name: Generate git-version.h - run: make version - - - name: Build pgaf:next (current branch, PG${{ env.PGVERSION }}) - run: make -C tests/upgrade pgaf-next PGVERSION=${{ env.PGVERSION }} - - - name: Build pgaf:current (previous release, PG${{ env.PGVERSION }}) - run: make -C tests/upgrade pgaf-current PGVERSION=${{ env.PGVERSION }} - - - name: Run upgrade spec - timeout-minutes: 25 - run: | - mkdir -p /tmp/pgaftest && chmod 777 /tmp/pgaftest - DOCKER_GID=$(stat -c '%g' /var/run/docker.sock) - docker run --rm \ - --user root \ - --group-add "${DOCKER_GID}" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /tmp/pgaftest:/tmp/pgaftest \ - -v "$(pwd)":/work:ro \ - -w /work \ - -e PGVERSION=${{ env.PGVERSION }} \ - -e PGAFTEST_HOST_WORK_DIR="$(pwd)" \ - pgaf:pgaftest \ - pgaftest run tests/tap/specs/upgrade.pgaf diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml deleted file mode 100644 index 322561aa4..000000000 --- a/.github/workflows/run-tests.yml +++ /dev/null @@ -1,101 +0,0 @@ -name: Run Tests - -on: - push: - branches: - - main - pull_request: - branches: - - main - - workflow_dispatch: - -jobs: - style_checker: - name: Style check - runs-on: ubuntu-latest - container: citus/stylechecker:no-py - steps: - - name: Checkout repository - uses: actions/checkout@v7.0.0 - - - name: Set safe directory for git - run: git config --global --add safe.directory ${GITHUB_WORKSPACE} - - - name: Check C formatting - run: citus_indent --check - - - name: Check banned functions - run: ci/banned.h.sh - - build_images: - name: Build test image (PG${{ matrix.PGVERSION }}) - needs: style_checker - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - PGVERSION: - - 14 - - 15 - - 16 - - 17 - steps: - - name: Checkout repository - uses: actions/checkout@v7.0.0 - - - name: Build Docker test image - run: PGVERSION=${{ matrix.PGVERSION }} make build-test-image - - - name: Save Docker image to archive - run: PGVERSION=${{ matrix.PGVERSION }} make save-test-image - - - name: Upload image archive - uses: actions/upload-artifact@v7.0.1 - with: - name: pg-test-image-${{ matrix.PGVERSION }} - path: pg_auto_failover_test-pg${{ matrix.PGVERSION }}.tar.zst - retention-days: 1 - - run_tests: - name: Run test (${{ matrix.PGVERSION }}, ${{ matrix.TEST }}) - runs-on: ubuntu-latest - needs: build_images - strategy: - fail-fast: false - matrix: - PGVERSION: - - 14 - - 15 - - 16 - - 17 - TEST: - - multi - - single - - monitor - - ssl - - citus - include: - - PGVERSION: 14 - TEST: tablespaces - steps: - - name: Checkout repository - uses: actions/checkout@v7.0.0 - - - name: Download image archive - uses: actions/download-artifact@v8.0.1 - with: - name: pg-test-image-${{ matrix.PGVERSION }} - - - name: Load Docker test image - run: PGVERSION=${{ matrix.PGVERSION }} make load-test-image - - - name: Set environment variables - run: | - echo "PGVERSION=${{ matrix.PGVERSION }}" >> $GITHUB_ENV - echo "TEST=${{ matrix.TEST }}" >> $GITHUB_ENV - echo "TRAVIS_BUILD_DIR=$(pwd)" >> $GITHUB_ENV - - - name: Run Test - timeout-minutes: 15 - run: make run-test-prebuilt From a4651f63c5f08d0efcc82f35a1fe76a08888a168 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sun, 12 Jul 2026 16:23:30 +0200 Subject: [PATCH 2/2] =?UTF-8?q?ci:=20fix=20base-images.yml=20=E2=80=94=20m?= =?UTF-8?q?atrix=20context=20not=20available=20in=20job-level=20if?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit matrix.* is only resolved within steps, not at the job-level if condition. Move the manual pgversion filter into the detect step instead, using an early exit so all downstream steps (gated on rebuild == 'true') are skipped naturally. Also drop the now-redundant PGVERSION_STR matrix field. --- .github/workflows/base-images.yml | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/.github/workflows/base-images.yml b/.github/workflows/base-images.yml index edb81fd91..bbf2e4dcc 100644 --- a/.github/workflows/base-images.yml +++ b/.github/workflows/base-images.yml @@ -90,10 +90,7 @@ jobs: name: Check minor releases (PG${{ matrix.PGVERSION }}) if: > github.event_name == 'schedule' && github.event.schedule == '0 3 * * 1' || - github.event_name == 'workflow_dispatch' && ( - github.event.inputs.pgversion == '' || - github.event.inputs.pgversion == matrix.PGVERSION_STR - ) + github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest permissions: contents: write @@ -102,11 +99,11 @@ jobs: fail-fast: false matrix: include: - - { PGVERSION: 14, PGVERSION_STR: '14', CITUSTAG: v12.1.5 } - - { PGVERSION: 15, PGVERSION_STR: '15', CITUSTAG: v12.1.5 } - - { PGVERSION: 16, PGVERSION_STR: '16', CITUSTAG: v13.2.0 } - - { PGVERSION: 17, PGVERSION_STR: '17', CITUSTAG: v13.2.0 } - - { PGVERSION: 18, PGVERSION_STR: '18', CITUSTAG: v14.1.0 } + - { PGVERSION: 14, CITUSTAG: v12.1.5 } + - { PGVERSION: 15, CITUSTAG: v12.1.5 } + - { PGVERSION: 16, CITUSTAG: v13.2.0 } + - { PGVERSION: 17, CITUSTAG: v13.2.0 } + - { PGVERSION: 18, CITUSTAG: v14.1.0 } steps: - uses: actions/checkout@v4 @@ -117,6 +114,14 @@ jobs: - name: Detect latest Postgres ${{ matrix.PGVERSION }} minor release id: detect run: | + # When triggered manually with a specific pgversion, skip other versions. + INPUT="${{ github.event.inputs.pgversion }}" + if [ -n "${INPUT}" ] && [ "${INPUT}" != "${{ matrix.PGVERSION }}" ]; then + echo "rebuild=false" >> $GITHUB_OUTPUT + echo "PG${{ matrix.PGVERSION }}: skipped (manual run requested PG${INPUT})" + exit 0 + fi + LATEST=$(git ls-remote --tags https://github.com/postgres/postgres \ "refs/tags/REL_${{ matrix.PGVERSION }}_*" \ | grep -v '\^{}' \