Skip to content

Add PostgreSQL 19 support, drop PostgreSQL 16 - #423

Open
ibrahim halatci (ihalatci) wants to merge 3 commits into
developfrom
ihalatci-didactic-invention
Open

Add PostgreSQL 19 support, drop PostgreSQL 16#423
ibrahim halatci (ihalatci) wants to merge 3 commits into
developfrom
ihalatci-didactic-invention

Conversation

@ihalatci

Copy link
Copy Markdown
Contributor

Adds PostgreSQL 19 support and drops PostgreSQL 16 across the packaging automation, in lockstep with the corresponding citusdata/docker change.

This is the tools-side half of the PG19 rollout. The packaging-side work is already merged (citusdata/packaging #1210#1213) and postgresql-19-hll 2.21.citus-1 / postgresql-19-topn 2.7.1.citus-1 are published and signed.

PG19 support / PG16 removal

  • update_docker.pySupportedDockerImages becomes postgres17=4, postgres18=5, postgres19=6; template/output maps drop postgres-16/* and add postgres-19/*; update_docker_file_for_postgres16() replaced with update_docker_file_for_postgres19(); update_all_docker_files() now derives latest from postgres_19_version and calls 17 → 18 → 19; read_postgres_versions() reads 19/18/17.
  • publish_docker.pyDockerImageType renumbered to latest=1, alpine=2, nightly=3, postgres_17=4, postgres_18=5, postgres_19=6, with a pg19 tag on ScheduleType.regular.
  • Templates — new postgres-19/postgres-19.tmpl.dockerfile; deleted postgres-14/, postgres-15/, postgres-16/ (all EOL). Extension pins moved to hll 2.21.citus-1 / topn 2.7.1.citus-1 in latest, postgres-17, postgres-18.
  • test-images/ — the 8 live distros bumped to hll 2.21.citus-1 / topn 2.7.1.citus-1. The 8 EOL distro images are deliberately untouched.
  • .github/workflows/package-tests.yml — fallback version 14.1.015.0.0.

Three bug fixes found along the way

1. update_docker.py:256git add --update .git add -A .

--update only stages files git already tracks, so a newly generated postgres-<N>/ directory was silently never committed. This is why postgres-18/Dockerfile and the pg18 Docker Hub tag do not exist today — release commit 75bc0c2 omitted the file. Without this fix, adding PG19 would reproduce the same failure.

2. citus_package.py:332 — nightly exclusions iterated the wrong list

nightly_versions = [
    v
    for v in release_versions      # was: release_versions, should be: nightly_versions
    if v not in exclude_dict_nightly[platform_key_nightly]
]

The block is currently unreachable because nightly: {} is empty and falsy, so this is latent — but the moment any nightly exclusion is added, nightly_versions gets rebuilt from the already-release-filtered list and the platform silently builds the wrong PG set. Verified against the real resolver: the only behavior change is ol/7 nightly ["13","14"]["14"], which is the correct result. test_get_postgres_versions_ol_7 was asserting the buggy value and has been corrected.

3. latest / alpine templates had drifted from citusdata/docker

The templates are the source of truth, so every automated release was silently reverting manual fixes made directly in the docker repo. A render-and-compare against the live repo showed postgres-17/18/19 matching but latest and alpine diverging. Resynced:

  • alpine.tmpl.dockerfileclang/llvmclang21/llvm21
  • latest.tmpl.dockerfile — back-ported the hadolint hardening from docker b47008b (SHELL with pipefail, version-pinned ca-certificates=*/curl=*, --no-install-recommends, merged the standalone RUN sed into the preceding RUN)

After the fix, both templates render byte-identical to the docker repo.

Validation

  • pytest -k "latest or alpine or postgres17 or postgres18 or postgres19" + test_publish_docker.py7 passed
  • Full render of all five templates in WSL → all five outputs correct
  • Cross-repo diff against live citusdata/dockerlatest, alpine, postgres-17/18/19 all MATCH

Known temporary failure

test_pkgvar_postgres_version_existence fails with KeyError: 'postgres_19_version' until the companion citusdata/docker PR merges — the test clones the live docker repo and reads its pkgvars, which does not define postgres_19_version yet. Expected, and resolves on its own once docker lands.

Sequencing

This PR must merge and be tagged before the citusdata/docker PG19 change can land — that change puts postgres_19 in the workflow matrix, which fails against any tools tag up to and including v0.8.38 (still postgres_16/17/18). Conversely, if a docker release runs with old tools against the new docker repo, it recreates postgres-16/ and KeyErrors on the missing postgres_16_version. The two need to go together.

ihalatci-msft and others added 3 commits August 20, 2026 12:39
Retarget the Docker image automation to the PG 17/18/19 window for
Citus 15.0.0, and bump the hll/topn extension pins to 2.21.citus-1 /
2.7.1.citus-1 across the docker templates and the live test images.

- update_docker.py / publish_docker.py: postgres16 -> postgres19
- templates/docker: add postgres-19, delete postgres-14/15/16
- latest.tmpl.dockerfile: bump stale hll 2.17 / topn 2.5.0 pins, which
  would otherwise break the `latest` image now that it tracks PG19
- test-images: bump hll/topn on the 8 non-EOL distros
- package-tests.yml: 14.1.0 -> 15.0.0

Also fixes two latent bugs that block the PG19 rollout:

1. update_docker.py used `git add --update .`, which stages only tracked
   files, so a newly generated image directory was never committed. This
   is why postgres-18/ and the pg18 Docker Hub tag do not exist today
   despite the script generating them. Now uses `git add -A .`.

2. citus_package.py built the nightly PG list by filtering
   `release_versions` instead of `nightly_versions`, so any nightly
   exclusion would silently emit release versions. Dormant only because
   the nightly exclusion map is currently empty; packaging is about to
   populate it for el/8 on PG19. The ol/7 unit test asserted the buggy
   output and has been corrected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
The latest and alpine templates had drifted behind the docker repo, so
every release run silently reverted manual fixes made there:

- alpine: clang/llvm -> clang21/llvm21 (c9917b8, re-applied in 75bc0c2).
  Alpine has no unversioned llvm package, so reverting this would break
  the PG19 alpine build.
- latest: back-port the hadolint hardening from b47008b (SHELL pipefail,
  pinned =* apt specs, --no-install-recommends, quoted package specs,
  sed merged into the chmod RUN).

Both templates now render byte-identical to the committed docker files.
The latest template gained one line, so its line-index assertions in
test_update_docker.py were adjusted accordingly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
Two CI regressions from this branch:

1. dbe5abd synced the alpine docker template with citusdata/docker,
   bumping clang/llvm to clang21/llvm21. test_process_template_file
   compares the rendered template byte-for-byte against
   expected_alpine_10.0.3.txt, so the fixture needs the same change.

2. The package-tests.yml fallback was bumped from 14.1.0 to 15.0.0,
   but Citus 15.0.0 is not released yet, so every test_execution job
   failed on "No match for argument: citus150_<pg>-15.0.0.citus-1".
   Reverted to 14.1.0. The bump is not needed for PostgreSQL 19
   support: 15.0.0 currently resolves to the same [16, 17, 18] matrix
   as 14.1.0, because packaging's postgres-matrix.yml has no 15.0
   entry yet. Both should be updated together when Citus 15.0 ships.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants