fix(ansible): move pgBackRest spool-path off the 10GB root volume - #2312
Open
hunleyd wants to merge 12 commits into
Open
Conversation
spool-path was never set explicitly, so pgBackRest defaulted to /var/spool/pgbackrest, which lands on the AMI's root volume (10GB, shared with the OS/systemd/journal) rather than the /data EBS volume PGDATA lives on. archive-get's async replica-catch-up queue can hold several 16MB+ WAL segments at once; once archive-async is enabled fleet-wide (INDATA-996) that risks filling the root volume and taking the instance down. Points spool-path at /data/pgbackrest_spool instead, and updates the directory-creation task to match. Global option, set once at provisioning — no supabase-admin-agent changes needed.
…est.yml The pgbackrest spool directory path is now split across two files (pgbackrest.conf's spool-path and this task's dir-creation loop); add a why-comment so a future edit to one doesn't silently drift from the other.
This comment has been minimized.
This comment has been minimized.
…nces The comment explaining the /data spool-path override packed the AMI root-volume constraint and the dedicated-EBS-volume fact into one long clause. Split into two short sentences on the same line per ASD-STE100 style rules.
Collaborator
|
Looks good. Will need to make sure it migrates correctly on existing instances. I think we need to move the directory if it exists. It depends on how/when pg_backrest is run and what is in the directory. |
…c-spool-path-defaults-onto-the-10gb-root-volume * origin/develop: (26 commits) Lots of bootstrap script clean ups (#2326) ci: Fix nix-build PUSH_TO_CACHE (#2345) ci/nix-install-ephemeral: Drop sticky disk config (#2346) chore: bump postgres_release to cut fresh AMIs (includes #2334) (#2349) fix(multigres): stop base config data_directory from overriding pooler data dir (#2344) fix(cron): remove unnecessary TRIGGER grant on cron.job_run_details from postgres (#2334) chore: enable extension version restriction (warn) and cut AMIs (#2315) feat(nix): add site-env packages for rolling instance updates (#2283) ci: resolve package install sha from a flake input override (#2327) fix(multigres): remove dangling wal-g include from postgresql.conf (#2338) fix: Allow test CI pipelines to use old Int CA access. For staging only (#2330) chore(nix): remove maintainers field from package definitions (#2280) chore: bump multigres to b713432 (#2323) ci: Use arm-native-runner for kvm builds on aarch64-linux (#2319) Whole lot of ansible clean up (#2272) fix(ansible): drop no-op zpool loop item from zswap task (#2322) ci: Fix dockerhub-release-matrix matrix generation (#2320) chore: bump pgctld (#2318) fix(ansible): skip zswap params the kernel does not expose (#2321) feat(docker): add Dockerfile-supabase base image and rewrite Dockerfile-multigres as layered image (#2160) ...
sbpostgres_apparmor's postgres_shell and pgbackrest_shell sub-profiles still only allowed writes to /var/spool/pgbackrest, the path this PR moved off of. Once archive-async is enabled, pgbackrest's writes to /data/pgbackrest_spool would be denied under AppArmor enforce mode, turning this fix into a new archive-push/archive-get failure.
…ool directory task
…in pgbackrest_spool comment
…c-spool-path-defaults-onto-the-10gb-root-volume * origin/develop: Report final AMI disk usage to workflow job (#2384) Re-organize build scripts, add to shellcheck (#2380) feat: Do not hardcode `orioledb.main_buffers` in `postgresql.conf` (#2365) feat: add orioledb debug symbols (#2379) chore: Make postgresql.env optional in database-optimizations.service (#2372) feat: bump to PostgREST v14.17 (#2374) ansible: Bump versions to get new AMIs built (#2376) Revert PostgREST bump to v14.17 (#2371) fix: disable APT timers during Supabase upgrades (#2358) gha/nix-install-ephemeral: Actually copy to s3, not just echo it (#2369) feat: bump to PostgREST v14.17 (#2363) feat: add guest-to-host time synchronisation (#2364) ci: Add final job that depends on the important jobs (#2361) feat: Add ORIOLEDB_ENABLED into /etc/environment.d/postgresql.env (#2362) fix: harden post-upgrade vacuum analyze in pg_upgrade complete script (#2356) feat: Bump admin-api to 0.114.0 (#2342) feat: [ansible/tasks] adding reindex script for collation version change (#2343) Revert "Lots of bootstrap script clean ups (#2326)" (#2357) ci: Delete nix-eval workflow (#2347)
Two added comments narrated the defect being fixed and one contained a factually wrong ordering claim (setup-postgres.yml doesn't mount /data; it's mounted from fstab at boot) — flagged independently by two review voices. The AppArmor bare-dir comment referenced 'pattern above' positionally instead of by path.
hunleyd
marked this pull request as ready for review
August 22, 2026 18:17
hunleyd
enabled auto-merge
August 22, 2026 20:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pgBackRest's
spool-pathwas never set explicitly, so it defaulted to/var/spool/pgbackreston the AMI's 10GB root volume instead of/data(the dedicated EBS volume PGDATA lives on). This PR setsspool-path = /data/pgbackrest_spoolin the ansible-managed pgbackrest.conf and updates the AppArmor profile to allow writes there, so archive-push/archive-get transient data survives events (pause/restore, pg_upgrade, resize-compute) that wipe the root volume.Details
spool-path = /data/pgbackrest_spoolin the ansible-managed[global]pgbackrest.conf, and updates the directory-creation task to create that path instead.sbpostgresAppArmor profile (postgres_shellandpgbackrest_shellsub-profiles) to grant write access to/data/pgbackrest_spoolinstead of the old/var/spool/pgbackrest. The profile runs in enforce mode, so without this change the config change alone would just trade "spool fills the root volume" for "every archive-push/archive-get write gets denied."/datais the volume that survives pause/restore, pg_upgrade, resize-compute, and restore-to-new-project. Those events can discard the existing EC2 instance and its root volume, so anything written there is lost.[global]-scope pgBackRest option, set once at provisioning. Nosupabase-admin-agentchanges needed. Split out of feat(sudoers): let supabase-admin-agent invoke pgbackrest reconcile as root #2291 (sudoers grant forpgbackrest reconcile) since the two are unrelated concerns: one is a privilege-escalation grant, this one is a storage-location fix, and neither depends on the other.Why this is safe
spool-pathlive on the same filesystem aspg_wal, soarchive-getcan hand a WAL segment to Postgres with a rename instead of a copy./data/pgdata/pg_walis on/data, so this change satisfies that recommendation directly. That's the primary justification./var/spool/pgbackrestalso shared the 10GB root volume with the OS, systemd, and journal, so headroom there was already thin; moving off it removes one more consumer of that shared space. This trades that exposure for the spool queue now sharing/datawith PGDATA instead: a smaller volume of contention (pgBackRest'sarchive-get-queue-maxdefaults to a bounded 128MiB), but worth naming as the tradeoff being accepted rather than a pure win.spool-pathcontents aren't durable state.archive-pushrechecks each WAL segment against the repo andarchive-getrebuilds its queue on loss. No migration needed for spool contents.pgbackrest:postgres) already applied to every other entry in the same directory-creation loop.Known gap (tracked separately, not blocking this PR)
/dataitself is root:root 0755, so an instance whose/datavolume didn't come fresh from this AMI's own data snapshot (resize-compute, hibernation-resume, pg major upgrade) never gets/data/pgbackrest_spoolcreated, and the unprivilegedpgbackrestuser can't self-create it there.supabase-admin-agent'sreconcilecommand already backfills this viaEnsureSpoolDir()(supabase-admin-agent#120, currently draft pending its own dependencies). It runs every ~10 minutes via the existing Salt cycle, the same mechanism this codebase already uses to close AMI-provisioning gaps on the live fleet.archive-asyncis off everywhere (INDATA-996 is the fleet-wide enablement gate).supabase-admin-agent#120needs to land before INDATA-996 ships fleet-wide, not before this PR merges.sbpostgresAppArmor profile duplicates its file-permission rules between thepostgres_shellandpgbackrest_shellsub-profiles (pre-existing, not introduced by this PR), tracked as INDATA-1180 for a future#include-based dedup.Testing
ansible-playbook --syntax-checkpasses cleanly on the full playbook/data/pgbackrest_spoolexists withpgbackrest:postgresownership after provisioning, andarchive-get/archive-pushoperate against it under the updated AppArmor profileMisc
Resolves INDATA-1038
Internal-only entry: supabase/changelog#225