Skip to content

fix(postgresql): restore cnpg-backup credentials to bucket-owner user#71

Open
TheMeinerLP wants to merge 1 commit into
mainfrom
fix/rgw-bucket-ownership-hijack
Open

fix(postgresql): restore cnpg-backup credentials to bucket-owner user#71
TheMeinerLP wants to merge 1 commit into
mainfrom
fix/rgw-bucket-ownership-hijack

Conversation

@TheMeinerLP

Copy link
Copy Markdown
Collaborator

Summary

  • The 2026-07-13 StorageClass provisioner fix (619fde4) let every long-pending ObjectBucketClaim finally bind on 2026-07-14 — and each one hijacked ownership of its already-populated bucket from the app's own CephObjectStoreUser to a freshly minted obc-<ns>-<name>-<uuid> ghost user. 13 of 15 real buckets were affected (discovered via AccessDeniedExceptions in Tempo traces from an external Minecraft server's BlueMap plugin). Ownership has already been re-linked back to each app's static user live via radosgw-admin bucket link (out-of-band, not part of this diff — RGW admin state, not something git tracks).
  • feather-core-cluster-pg-backup (CNPG WAL archiving) was a special case: an earlier session (3d44843) had already worked around the same symptom by repointing cnpg-backup's SOPS secret at the ghost owner's own key pair — correct at the time, but now stale since bucket ownership points back at postgresql-cluster-backup. This PR switches the secret back to that static user's key pair so credentials and bucket ownership agree again.
  • Documents the whole incident, detection, and fix in docs/buckets.md, including the "two mutually exclusive fixes" trap this PR itself had to resolve, so it doesn't happen again.

Test plan

  • Verified via radosgw-admin bucket stats that all 15 affected buckets are now owned by their correct static CephObjectStoreUser.
  • Verified via Tempo traces that Survival-1/BlueMap's AccessDeniedExceptions stopped after the relink and a previously-failing PUT succeeded.
  • Verified via sops -d that s3-backup.env now decrypts to the postgresql-cluster-backup user's access key.
  • After merge, confirm flux reconcile kustomization configs --with-source picks up the new secret and CNPG's plugin-barman-cloud container stops logging AccessDenied (currently still failing on main since the secret isn't live yet).

🤖 Generated with Claude Code

The 2026-07-14 provisioner fix caused every long-pending ObjectBucketClaim
to bind and hijack ownership of its already-populated bucket from the
app's own CephObjectStoreUser to a freshly minted obc-<ns>-<name>-<uuid>
ghost user (see docs/buckets.md). 13 of 15 real buckets were affected;
bucket ownership has now been re-linked back to each app's static user via
radosgw-admin.

feather-core-cluster-pg-backup was a special case: an earlier session had
already worked around the AccessDenied by repointing cnpg-backup's SOPS
secret at the ghost owner's own key pair (3d44843), which was correct at
the time but is now stale now that ownership points back at
postgresql-cluster-backup. Switch the secret back to that static user's
key pair so credentials and bucket ownership agree again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants