cluster: encrypt and authenticate controller to replica connections

[jasonhernandez]

Motivation

PR 3 of 3. Turns on what #37995 (CTP mutual TLS support) and #37996 (internal secrets) built: controller↔replica traffic is encrypted and both ends authenticate each other.

Today CTP is plaintext bincode with no authentication. It carries user rows (peek results, SUBSCRIBE batches), and because a CTP server serves one client and cancels the incumbent on any new connect, anyone who can reach a replica's *ctl port can evict the controller and take over the command stream. The trust model is "the pod network is isolated," which stops holding once an operator can mirror ENI traffic.

Stacked on #37996 (which is stacked on #37995). Review those first; this branch merges both.

How it works

• environmentd, behind --cluster-transport-tls, bootstraps a per-environment CA into an internal secret at startup and mints its own controller certificate.
• The cluster controller mints a certificate per replica at ensure_replica_location time, writes it to the replica's internal secret, and passes --ctp-tls-secret=<name> to clusterd.
• clusterd reads the credentials through the SecretsReader it already links and requires mutual TLS on both controller listeners.
• Dropping a replica deletes its secret.

Identity is the replica's service name in the certificate SAN. The controller verifies it reached the intended replica; the replica verifies the peer holds a controller certificate from the same environment. Distribution rides the machinery that already exists for user secrets: no cert-manager, no operator changes, no new RBAC.

No shell required, by construction

This is meant to hold up the other half of the distroless bargain, so the credential lifecycle never needs an operator inside a container:

• Nothing out of process. CA generation, issuance, and loading are all in-process (rcgen + the secrets reader). No init containers, no sidecars, no openssl CLI, no scripts — including for rotation.
• Rotation rides process lifecycle. Replica certificates are re-minted on every replica process creation, so there is no fleet-wide renewal operation. The 5-year validity is a backstop, not the rotation mechanism.
• Self-healing. A missing or corrupt credential secret is repaired by the controller re-ensuring it; clusterd retries the read at boot rather than crash-looping, since the write races process creation.
• Diagnosable from outside. A per-server mz_cluster_server_tls_handshake_failures_total counter, an mz_ctp_tls_cert_not_after gauge, and handshake-failure logs naming the peer and reason. kubectl logs plus scraping is enough — no openssl s_client needed.

Behavior changes

• Displacement now happens after authentication. A new client displaces the established one only once it has passed transport security, so with TLS on, an unauthenticated peer reaching the port cannot interrupt the live connection. The handshake runs in its own task bounded at 30s, so a stalled peer blocks neither the accept loop nor the live connection. Without TLS, displacement is unchanged (accept-time). A turmoil test covers the new property.
• CTP now flushes after each message. A raw socket write reaches the peer on its own, but a TLS writer buffers plaintext into records and need not emit them until flushed. Without this, messages (including the CTP handshake) strand indefinitely. Found by running the real process orchestrator; the turmoil socket masks it.
• Unmanaged replicas connect in plaintext. They run outside the orchestrator, so there is no channel for distributing credentials.
• Enabled by default in the Rust test harness, sqllogictest, and mzcompose (version-gated for older images); production opts in explicitly.

Testing

• New turmoil test asserting an unauthenticated peer cannot displace an authenticated connection, on top of the nine TLS tests from service: add mutual TLS support to the cluster transport protocol #37995 (23 tests total in the CTP suite, run 15+ times with random seeds, no flakes).
• Verified end to end against the real process orchestrator: all 8 CTP connections (4 replicas × storage/compute) completed mutual TLS handshakes and builtin catalog dataflows were created over them — this is what surfaced the missing flush.
• Caveat, stated plainly: I could not get a full mz-environmentd integration test to pass locally — test_empty_subscribe_notice times out at 240s. I A/B'd it with cluster_tls forced to None and it times out identically, so the timeout is environmental (local debug build against a containerized CockroachDB) and not caused by this change. CI is the real signal for the integration suites; worth a careful look at the first run.

🤖 Generated with Claude Code