service: add mutual TLS support to the cluster transport protocol

[jasonhernandez]

Motivation

CTP (the controller↔replica transport) is plaintext bincode with no authentication. It carries user rows (peek results, SUBSCRIBE batches), and its single-client-server design means anyone who can reach a replica's *ctl port can displace the controller and take over the command stream. The trust model today is "the pod network is isolated", which breaks down once an operator can mirror ENI traffic, and it is a gap we want to close as part of removing shells and other operator access from the data plane.

This PR adds the transport-level building block: optional mutual TLS on CTP connections, with identity pinning on both sides. It is PR 1 of a planned stack of 3:

1. This PR: TLS support in mz_service::transport, plus CA/leaf issuance helpers. Nothing enables it; all callers pass None, so behavior is unchanged.
2. Name-keyed internal credential support in SecretsController/SecretsReader, so the controller can distribute per-replica credentials through the existing secrets machinery.
3. End-to-end wiring: environmentd bootstraps a per-environment CA, the controller mints a leaf per replica at ensure_replica_location time, clusterd requires TLS on its controller listeners, plus observability and rollout gating.

Design notes

• TLS 1.3 only, explicit aws-lc-rs provider. Configs are built with builder_with_provider, so this neither depends on nor installs a process-default provider, and stays compatible with the FIPS path.
• Identity is a DNS-shaped name in the certificate SAN. The client verifies the server through standard server-name verification. The server requires client certificates via WebPkiClientVerifier and checks the client's SAN against the expected identity after the handshake. The CTP Hello handshake is unchanged and runs inside the TLS channel, demoted to a compatibility check.
• CertificateAuthority issues a path-length-zero CA and per-endpoint leaves (ECDSA P-256), with PEM persistence and reconstruction. Reconstruction rebuilds signing state from the same deterministic params helper as generation, keyed by the common name, rather than parsing the certificate. This avoids rcgen's x509-parser feature (which drags in a banned lazy_static and a duplicate syn), and a name mismatch fails closed: issued leaves carry the wrong issuer name and cannot verify. A test proves the fail-closed behavior.
• Connection::start now takes pre-split stream halves so the same code runs on plain and TLS streams. The server bounds the TLS handshake at 30s so a stalled or non-TLS peer cannot occupy the single connection slot indefinitely.
• Key material is mz_ore::secure::SecureString (zeroed on drop, redacted in debug output, non-Clone).
• Connection displacement still happens at TCP accept time, as today. Moving displacement to after peer authentication (so only a valid controller can evict the incumbent) is part of PR 3, where behavior is allowed to change.

Cargo.lock note

Cargo.lock gains entries for x509-parser and its dependency chain even though the rcgen feature is not enabled: rcgen references it as a weak dependency feature (x509-parser?/verify-aws), and cargo's version resolver locks optional dependencies regardless of activation. cargo tree -i x509-parser confirms nothing depends on it and it is never built. cargo deny check licenses bans passes.

Testing

Extends the existing turmoil-based CTP test suite (src/service/tests/transport.rs) with nine TLS tests, all running under the same deterministic simulation as the existing tests:

• Happy path: bidirectional communication, keepalives, >1 MiB messages (TLS record chunking), connection cancelation/displacement under TLS.
• Negative: untrusted server CA, wrong server identity, untrusted client CA, wrong client identity, plaintext client vs TLS server, TLS client vs plaintext server, expired certificate, and CA-reconstruction name mismatch. All rejection tests assert client-side that the connection fails; a wrongly-established connection fails the test via expect_err.
• CA PEM round-trip: a CA reconstructed from persisted PEMs issues certificates that chain to the original CA certificate (the property PR 3's controller restart path relies on).

The suite was run 20+ times locally with random simulation seeds with no flakes. Existing tests updated mechanically for the new tls parameter.

🤖 Generated with Claude Code