Add PostgreSQL 19 support, drop PostgreSQL 16 - #423
Open
ibrahim halatci (ihalatci) wants to merge 3 commits into
Open
Add PostgreSQL 19 support, drop PostgreSQL 16#423ibrahim halatci (ihalatci) wants to merge 3 commits into
ibrahim halatci (ihalatci) wants to merge 3 commits into
Conversation
Retarget the Docker image automation to the PG 17/18/19 window for Citus 15.0.0, and bump the hll/topn extension pins to 2.21.citus-1 / 2.7.1.citus-1 across the docker templates and the live test images. - update_docker.py / publish_docker.py: postgres16 -> postgres19 - templates/docker: add postgres-19, delete postgres-14/15/16 - latest.tmpl.dockerfile: bump stale hll 2.17 / topn 2.5.0 pins, which would otherwise break the `latest` image now that it tracks PG19 - test-images: bump hll/topn on the 8 non-EOL distros - package-tests.yml: 14.1.0 -> 15.0.0 Also fixes two latent bugs that block the PG19 rollout: 1. update_docker.py used `git add --update .`, which stages only tracked files, so a newly generated image directory was never committed. This is why postgres-18/ and the pg18 Docker Hub tag do not exist today despite the script generating them. Now uses `git add -A .`. 2. citus_package.py built the nightly PG list by filtering `release_versions` instead of `nightly_versions`, so any nightly exclusion would silently emit release versions. Dormant only because the nightly exclusion map is currently empty; packaging is about to populate it for el/8 on PG19. The ol/7 unit test asserted the buggy output and has been corrected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
The latest and alpine templates had drifted behind the docker repo, so every release run silently reverted manual fixes made there: - alpine: clang/llvm -> clang21/llvm21 (c9917b8, re-applied in 75bc0c2). Alpine has no unversioned llvm package, so reverting this would break the PG19 alpine build. - latest: back-port the hadolint hardening from b47008b (SHELL pipefail, pinned =* apt specs, --no-install-recommends, quoted package specs, sed merged into the chmod RUN). Both templates now render byte-identical to the committed docker files. The latest template gained one line, so its line-index assertions in test_update_docker.py were adjusted accordingly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
Two CI regressions from this branch: 1. dbe5abd synced the alpine docker template with citusdata/docker, bumping clang/llvm to clang21/llvm21. test_process_template_file compares the rendered template byte-for-byte against expected_alpine_10.0.3.txt, so the fixture needs the same change. 2. The package-tests.yml fallback was bumped from 14.1.0 to 15.0.0, but Citus 15.0.0 is not released yet, so every test_execution job failed on "No match for argument: citus150_<pg>-15.0.0.citus-1". Reverted to 14.1.0. The bump is not needed for PostgreSQL 19 support: 15.0.0 currently resolves to the same [16, 17, 18] matrix as 14.1.0, because packaging's postgres-matrix.yml has no 15.0 entry yet. Both should be updated together when Citus 15.0 ships. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1ed7b58a-0ba9-4f47-a1de-3ee665bae7f2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds PostgreSQL 19 support and drops PostgreSQL 16 across the packaging automation, in lockstep with the corresponding
citusdata/dockerchange.This is the tools-side half of the PG19 rollout. The packaging-side work is already merged (citusdata/packaging #1210–#1213) and
postgresql-19-hll 2.21.citus-1/postgresql-19-topn 2.7.1.citus-1are published and signed.PG19 support / PG16 removal
update_docker.py—SupportedDockerImagesbecomespostgres17=4, postgres18=5, postgres19=6; template/output maps droppostgres-16/*and addpostgres-19/*;update_docker_file_for_postgres16()replaced withupdate_docker_file_for_postgres19();update_all_docker_files()now deriveslatestfrompostgres_19_versionand calls 17 → 18 → 19;read_postgres_versions()reads 19/18/17.publish_docker.py—DockerImageTyperenumbered tolatest=1, alpine=2, nightly=3, postgres_17=4, postgres_18=5, postgres_19=6, with apg19tag onScheduleType.regular.postgres-19/postgres-19.tmpl.dockerfile; deletedpostgres-14/,postgres-15/,postgres-16/(all EOL). Extension pins moved to hll2.21.citus-1/ topn2.7.1.citus-1inlatest,postgres-17,postgres-18.test-images/— the 8 live distros bumped to hll2.21.citus-1/ topn2.7.1.citus-1. The 8 EOL distro images are deliberately untouched..github/workflows/package-tests.yml— fallback version14.1.0→15.0.0.Three bug fixes found along the way
1.
update_docker.py:256—git add --update .→git add -A .--updateonly stages files git already tracks, so a newly generatedpostgres-<N>/directory was silently never committed. This is whypostgres-18/Dockerfileand thepg18Docker Hub tag do not exist today — release commit75bc0c2omitted the file. Without this fix, adding PG19 would reproduce the same failure.2.
citus_package.py:332— nightly exclusions iterated the wrong listThe block is currently unreachable because
nightly: {}is empty and falsy, so this is latent — but the moment any nightly exclusion is added,nightly_versionsgets rebuilt from the already-release-filtered list and the platform silently builds the wrong PG set. Verified against the real resolver: the only behavior change isol/7nightly["13","14"]→["14"], which is the correct result.test_get_postgres_versions_ol_7was asserting the buggy value and has been corrected.3.
latest/alpinetemplates had drifted fromcitusdata/dockerThe templates are the source of truth, so every automated release was silently reverting manual fixes made directly in the docker repo. A render-and-compare against the live repo showed
postgres-17/18/19matching butlatestandalpinediverging. Resynced:alpine.tmpl.dockerfile—clang/llvm→clang21/llvm21latest.tmpl.dockerfile— back-ported the hadolint hardening from dockerb47008b(SHELLwithpipefail, version-pinnedca-certificates=*/curl=*,--no-install-recommends, merged the standaloneRUN sedinto the precedingRUN)After the fix, both templates render byte-identical to the docker repo.
Validation
pytest -k "latest or alpine or postgres17 or postgres18 or postgres19"+test_publish_docker.py→ 7 passedcitusdata/docker→latest,alpine,postgres-17/18/19all MATCHKnown temporary failure
test_pkgvar_postgres_version_existencefails withKeyError: 'postgres_19_version'until the companioncitusdata/dockerPR merges — the test clones the live docker repo and reads itspkgvars, which does not definepostgres_19_versionyet. Expected, and resolves on its own once docker lands.Sequencing
This PR must merge and be tagged before the
citusdata/dockerPG19 change can land — that change putspostgres_19in the workflow matrix, which fails against any tools tag up to and includingv0.8.38(stillpostgres_16/17/18). Conversely, if a docker release runs with old tools against the new docker repo, it recreatespostgres-16/and KeyErrors on the missingpostgres_16_version. The two need to go together.