feat(nix): expose per-version extension packages, catalog, and update tools - #2317
Open
brainrake wants to merge 3 commits into
Open
feat(nix): expose per-version extension packages, catalog, and update tools#2317brainrake wants to merge 3 commits into
brainrake wants to merge 3 commits into
Conversation
PostgreSQL Package Dependency Analysis: PR #2317
SummaryNo packages had MAJOR version updates. Full Analysis ResultsPostgreSQL 15 Dependency ChangesExtracting PostgreSQL 15 dependencies...
Runtime Closure Size
Raw Dependency ClosurePostgreSQL 17 Dependency ChangesExtracting PostgreSQL 17 dependencies...
Runtime Closure Size
Raw Dependency Closure |
PostgreSQL Extension Dependency Analysis: PR #2317
SummaryNo extensions had dependencies with MAJOR version updates. Full Analysis ResultsPostgreSQL 15 Extension DependenciesPostgreSQL 17 Extension DependenciesOrioleDB 17 Extension Dependencies |
brainrake
changed the base branch from
martonboros/psql-1530-expose-ext-perversion
to
develop
July 27, 2026 18:26
brainrake
changed the title
brainrake
force-pushed
the
martonboros/psql-1530-ext-catalog
branch
3 times, most recently
from
July 27, 2026 19:46
d11cd58 to
998a591
Compare
brainrake
changed the title
brainrake
force-pushed
the
martonboros/psql-1530-ext-catalog
branch
from
July 30, 2026 15:15
3fe60a2 to
998a591
Compare
brainrake
marked this pull request as ready for review
July 31, 2026 02:01
brainrake
force-pushed
the
martonboros/psql-1530-ext-catalog
branch
from
August 2, 2026 00:24
d2144e3 to
e64f340
Compare
brainrake
enabled auto-merge
August 2, 2026 00:26
…ce tools
Adds passthru.perVersion (version -> single-version derivation) to each
multi-version extension (18 files, one line each) -- the attrset already
existed inline as the buildEnv input, this surfaces it instead of
discarding it. No change to any buildEnv output or extension identity.
New flake-parts module nix/packages/extension-catalog.nix, exposing
(packages + legacyPackages):
- site-extensions-catalog-<major> (15/17/orioledb-17): pg-extensions-
catalog.json mapping { ext -> { extversion -> store path } }, plus
bin/{resolve,update} preset with PG_EXTENSIONS_CATALOG. Assembled at
build time from the wrappers' control files, so keys are real
extversions (pgsql-http nix "1.5.0" -> "1.5") and include
sub-extensions (postgis_topology, plcoffee, ...). unsafeDiscardReferences
keeps the closure to just the JSON (6.8 KiB, not 2.4 GiB).
- site-extensions-versions-<major> (legacyPackages): the generic
single-version packages the catalog points at (lean, no multi-version
buildEnv), so the CI matrix builds and caches them.
- Generic tools, one version each (not per-major): site-extensions-resolve
(manifest json -> store paths), download-nix-store-paths (retry/timeout
substitution, not extension-specific), site-extensions-update (chains
both + nix-env --install --remove-all onto PROFILE, atomic).
Part of PSQL-1530.
brainrake
force-pushed
the
martonboros/psql-1530-ext-catalog
branch
from
August 24, 2026 15:02
e64f340 to
5bfbb75
Compare
mmlb
requested changes
Aug 24, 2026
mmlb
left a comment
mmlb
left a comment
Collaborator
There was a problem hiding this comment.
A few nits (bashisms and quotes) that would be nice to fix up but the big one to me is the [[ -n $name ]] || continue that doesn't smell good to me. Also the serial vs parallel download.
| b=$(basename "$cf") | ||
| n="''${b%%--*}" | ||
| v="''${b#*--}"; v="''${v%.control}" | ||
| [ -e "$ext/$n.control" ] && continue |
Collaborator
There was a problem hiding this comment.
[[ ]] is a bash builtin and better than [ command, we should prefer it since its a bash script anyway.
Suggested change
| [ -e "$ext/$n.control" ] && continue | |
| [[ -e "$ext/$n.control" ]] && continue |
Comment on lines
+28
to
+29
| n="''${b%%--*}" | ||
| v="''${b#*--}"; v="''${v%.control}" |
Collaborator
There was a problem hiding this comment.
Don't need to use " in assignment and is easier to read with the nix escaping in there too.
Suggested change
| n="''${b%%--*}" | |
| v="''${b#*--}"; v="''${v%.control}" | |
| n=''${b%%--*} | |
| v=''${b#*--}; v=''${v%.control} |
Comment on lines
+37
to
+42
| suffix=".''${b##*.}" | ||
| core="''${b%.*}" | ||
| base="''${core%-[0-9]*}" | ||
| [ "$base" != "$core" ] || continue | ||
| gen="$base$suffix" | ||
| [ -e "$out/lib/$gen" ] || ln -sfn "$b" "$out/lib/$gen" |
Collaborator
There was a problem hiding this comment.
ditto re " and [[ ]] bashism
Suggested change
| suffix=".''${b##*.}" | |
| core="''${b%.*}" | |
| base="''${core%-[0-9]*}" | |
| [ "$base" != "$core" ] || continue | |
| gen="$base$suffix" | |
| [ -e "$out/lib/$gen" ] || ln -sfn "$b" "$out/lib/$gen" | |
| suffix=.''${b##*.} | |
| core=''${b%.*} | |
| base=''${core%-[0-9]*} | |
| [[ "$base" != "$core" ]] || continue | |
| gen=$base$suffix | |
| [[ -e "$out/lib/$gen" ]] || ln -sfn "$b" "$out/lib/$gen" |
| esac | ||
| done | ||
| libs=$(ls -A "$out/lib" 2>/dev/null | wc -l) | ||
| if [ "$generic" -eq 0 ] && [ "$libs" -eq 0 ]; then |
Collaborator
There was a problem hiding this comment.
bash's math tests are nicer than -eq, -ne ...
Suggested change
| if [ "$generic" -eq 0 ] && [ "$libs" -eq 0 ]; then | |
| if ((generic == 0)) && ((libs == 0)); then |
Comment on lines
+88
to
+90
| name="''${base%.control}" | ||
| ver=$(sed -n "s/^default_version = '\(.*\)'.*/\1/p" "$ctrl" | head -1) | ||
| [ -n "$ver" ] || { echo "no default_version in $ctrl" >&2; exit 1; } |
Collaborator
There was a problem hiding this comment.
same thing re bashism and quotes. The case could be swapped out for [[ $base == *--* ]] instead but meh.
Suggested change
| name="''${base%.control}" | |
| ver=$(sed -n "s/^default_version = '\(.*\)'.*/\1/p" "$ctrl" | head -1) | |
| [ -n "$ver" ] || { echo "no default_version in $ctrl" >&2; exit 1; } | |
| name=''${base%.control} | |
| ver=$(sed -n "s/^default_version = '\(.*\)'.*/\1/p" "$ctrl" | head -1) | |
| [[ -n "$ver" ]] || { echo "no default_version in $ctrl" >&2; exit 1; } |
| : "''${PG_EXTENSIONS_CATALOG:?PG_EXTENSIONS_CATALOG must point at a pg-extensions-catalog.json}" | ||
| manifest="''${1:?Usage: $0 path-to/pg-extensions.json}" | ||
| jq -r 'to_entries[] | "\(.key)=\(.value)"' "$manifest" | while IFS='=' read -r name version; do | ||
| [ -n "$name" ] || continue |
Collaborator
There was a problem hiding this comment.
this seems weird, why would we end up with an empty line? If valid, worth a comment for why we might be seeing empty lines
| ln -s ${mkCatalogFile wrappers}/share/pg-extensions-catalog.json "$out/share/" | ||
| makeWrapper ${self'.packages.site-extensions-resolve}/bin/site-extensions-resolve \ | ||
| "$out/bin/site-extensions-resolve" \ | ||
| --set PG_EXTENSIONS_CATALOG "$out/share/pg-extensions-catalog.json" |
Collaborator
There was a problem hiding this comment.
why not --prefix instead? I'd be nice to not have to specify both an envvar and positional arg to the unwrapped scripts, better to both the same way imo.
| text = '' | ||
| manifest="''${1:-/etc/adminapi/pg-extensions.json}" | ||
| profile="''${PROFILE:-/nix/var/nix/profiles/site-extensions}" | ||
| mapfile -t paths < <(site-extensions-resolve "$manifest") |
Collaborator
There was a problem hiding this comment.
We have 2 readarrays and 0 mapfiles in the tree so good to use the same name, not to mention that readarray is more obvious than mapfile imo.
Suggested change
| mapfile -t paths < <(site-extensions-resolve "$manifest") | |
| readarray -t paths < <(site-extensions-resolve "$manifest") |
| done | ||
| echo "ERROR: failed to realize $path" >&2 | ||
| exit 1 | ||
| done |
Collaborator
There was a problem hiding this comment.
The loop as written makes for serial downloads instead of letting nix download in parallel, if network is flaky this is going to be pretty terrible experience waiting 10s for each path * 3. Maybe makes the happy path a little slower too (but probably not a big deal).
I'd probably do something like
ok=false
for _ in 1 2 3 4; do
timeout -k 10s 120s nix-store -r "$@" || continue
ok=true
break
done
if ! $ok; then
ok=true
for path in "$@"; do
echo timeout -k 10s 120s nix-store -r "$path" >/dev/null
echo "ERROR: nix failed to fetch $path after 5 attempts" >&2
ok=false
done
fi
if ! $ok;
exit 1
fi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of PSQL-1530.
Adds
passthru.perVersion(version -> single-version derivation) to each multi-version extension (18 files, one line each).Adds flake module
nix/packages/extension-catalog.nix, exposingpackagesandlegacyPackages:site-extensions-catalog-<major>(15/17/orioledb-17)pg-extensions-catalog.jsonmapping{ ext -> { extversion -> store path } }bin/{resolve,update}that use said jsonsite-extensions-versions-<major>: single-version ext packages the catalog points at (lean, ~30 MiB closures, no multi-version buildEnv), so CI builds and caches them.site-extensions-resolve(manifest json -> store paths, readsPG_EXTENSIONS_CATALOG)download-nix-store-paths(retry/timeout substitution, not extension-specific)site-extensions-update(chains both +nix-env --install --remove-allontoPROFILE, atomic — nix-env only swaps generations on success)Should be safe since it only adds new outputs.