Skip to content

Fix fresh-install failures in the mirror playbooks (task ordering, SELinux, certbot failure handling) - #5

Open
georgetasioulis wants to merge 2 commits into
cloudlinux:mainfrom
georgetasioulis:fix/playbook-selinux-and-ordering
Open

Fix fresh-install failures in the mirror playbooks (task ordering, SELinux, certbot failure handling)#5
georgetasioulis wants to merge 2 commits into
cloudlinux:mainfrom
georgetasioulis:fix/playbook-selinux-and-ordering

Conversation

@georgetasioulis

@georgetasioulis georgetasioulis commented Aug 5, 2026

Copy link
Copy Markdown

Summary

While standing up a new SWNG mirror on a fresh AlmaLinux 10 server with ansible/complete-swng-rsync, I hit five issues that prevent the playbooks from completing (or silently break syncing afterwards). All of them exist in the other playbook variants too, so this PR fixes them across complete-swng-rsync, combined-mirror, specific-version-rsync(Recomended), and yum-reposync.

1. Disk-space check runs before the mirror directory exists

The first task runs df -h {{ mirror_base_path }}, but the directory is only created a few tasks later — on a fresh server the play aborts immediately with df: /var/www/mirrors: No such file or directory. Fixed by creating the directory before the check.

2. SELinux relabel is not recursive

The relabel task adds the public_content_rw_t fcontext rule but runs restorecon non-recursively on /var/www only. Subdirectories — including the ACME webroot — keep their var_t label, so nginx returns 403 for Let's Encrypt HTTP-01 challenges and certbot can never obtain a certificate. Re-running the playbook doesn't help since the relabel repeats non-recursively. Fixed with restorecon -RFv.

3. rsync fails under systemd on enforcing SELinux (exit code 10)

systemd-spawned rsync transitions into the confined rsync_t domain, which by default cannot make outbound connections:

type=AVC msg=audit(...): avc:  denied  { name_connect } for  pid=31169 comm="rsync"
  dest=873 scontext=system_u:system_r:rsync_t:s0
  tcontext=system_u:object_r:rsync_port_t:s0 tclass=tcp_socket permissive=0

Every timer-triggered sync dies instantly (rsync error: error in socket IO (code 10)), while the Ansible-invoked initial sync works because it runs unconfined — masking the bug until the first timer firing. Fixed by setting the rsync_client and rsync_anon_write booleans in the rsync-based playbooks (not needed for yum-reposync, which doesn't run in rsync_t).

4. HTTPS vhost deployed even when certbot failed

The certbot tasks use ignore_errors: yes, but the HTTPS vhost template is deployed unconditionally afterwards. When certificate issuance fails (e.g. because of issue 2), nginx is pointed at nonexistent certificate files and fails to start entirely — which in turn blocks any webroot-based retry, leaving the server in a state the playbook cannot recover from. Fixed by gating the HTTPS vhost (and the HTTP→HTTPS redirect) on the certificate actually existing, with a visible warning when it is skipped.

5. Confined rsync cannot write the upstream-owned tree (silent no-op syncs)

With issue 3 fixed, syncs connect but still fail: rsync -a preserves upstream ownership, so the tree is owned by non-root uids (e.g. 11122/11373) with mode 775 directories. The confined rsync_t receiver runs as root but is not granted the dac_override capability, so every mkstemp fails with Permission denied (13) — and the denial is dontaudited, so the audit log stays empty while the sync grinds through the entire file list writing nothing:

type=AVC msg=audit(...): avc:  denied  { dac_override } for  comm="rsync" capability=1
  scontext=system_u:system_r:rsync_t:s0 tclass=capability permissive=0

(captured with semodule -DB). The rsync_full_access boolean does not help — on AlmaLinux 10 it grants file-access rules but not the capability.

Rather than granting rsync_t the dac_override capability (broader than the problem requires), the fix stops preserving upstream ownership, which has no value on a mirror that only serves files over HTTP: --no-owner --no-group is added to the initial sync commands and every systemd service template, so the tree is root-owned and confined rsync needs no extra capability. Existing deployments synced with the old owner-preserving flags are normalized once via a find-guarded recursive chown.

Testing

  • Hit all five issues on a real deployment (fresh AlmaLinux 10.2, complete-swng-rsync); verified each fix resolves the corresponding failure and the full playbook now converges: cert issued, HTTPS serving, timer-driven syncs running confined under rsync_t and writing the tree. For issue 5, verified both directions on the same server: the confined service fails on the upstream-owned tree and writes normally after chown -R root:root.
  • ansible-playbook --syntax-check passes for all four modified playbooks.

🤖 Generated with Claude Code

Four issues that break the playbooks on a fresh AlmaLinux 10 server:

1. Disk-space check ran before the mirror directory was created,
   aborting the play on first run (df on a nonexistent path).
   Reordered directory creation before the check.

2. The SELinux relabel used a non-recursive restorecon, leaving
   subdirectories (including the ACME webroot) labeled var_t. Nginx
   then returned 403 for Let's Encrypt HTTP-01 challenges and no
   certificate could ever be issued. Now relabels recursively.

3. rsync run from the systemd service is confined to rsync_t, which
   cannot make outbound connections by default (AVC name_connect
   denial on port 873, rsync exit code 10). Every timer-triggered
   sync failed instantly, while manual/Ansible-invoked rsync worked
   because it runs unconfined - masking the bug until the timer
   fired. Set the rsync_client and rsync_anon_write booleans in the
   rsync-based playbooks.

4. The HTTPS vhost was deployed even when certbot failed (its
   failure is ignored), pointing nginx at nonexistent certificate
   files. nginx then failed to start at all, which also blocks any
   webroot retry - a chicken-and-egg state requiring manual
   recovery. The HTTPS config is now gated on the certificate
   actually existing, with a warning when it is skipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@georgetasioulis
georgetasioulis force-pushed the fix/playbook-selinux-and-ordering branch from aa8451d to ff0ce56 Compare August 6, 2026 00:00
With the rsync_client boolean set, timer-driven syncs connect but
still fail to write: rsync -a preserves upstream ownership, so the
tree ends up owned by non-root uids (e.g. 11122/11373) with mode 775
directories. The systemd-run rsync receiver is confined to rsync_t,
which lacks the dac_override capability, so root cannot create files
in directories it does not own:

  avc: denied { dac_override } for comm="rsync" capability=1
    scontext=system_u:system_r:rsync_t:s0 tclass=capability permissive=0

The denial is dontaudited by default, so the audit log stays empty
(visible only with semodule -DB) while every mkstemp fails EACCES and
the sync grinds through the whole file list writing nothing. The
rsync_full_access boolean does not help - on AlmaLinux 10 it grants
file access rules but not the capability.

Upstream UID/GID values have no meaning on a mirror that only serves
files over HTTP, so stop preserving them instead of widening policy:

- Add --no-owner --no-group to the initial sync commands and every
  systemd service template; transferred files are then owned by the
  receiving user (root) and confined rsync_t writes root-owned
  directories without any extra capability.
- Normalize existing trees once (guarded by a find probe so the
  recursive chown only runs when non-root-owned files exist, e.g. on
  deployments synced with the previous owner-preserving flags).

Verified on a real AlmaLinux 10.2 deployment: the confined service
fails on the upstream-owned tree and writes normally after
chown -R root:root.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@georgetasioulis
georgetasioulis force-pushed the fix/playbook-selinux-and-ordering branch from ff0ce56 to f6fbf32 Compare August 6, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant