[control] QUIC/mTLS key exchange + PSP-model data plane (APO-648/APO-644)#1
Merged
Conversation
…ad guard (APO-647) Two fixes to both decap paths (PhyToVirt and PhyToVirtInPlace): APO-645 / S2 — verify the AEAD tag BEFORE touching the replay window. ValidateCounter both checks and ADVANCES the sliding window, and it was running before rxCipher.Open since the initial commit. That let an off-path attacker who can spoof the outer 4-tuple advance the window with a forged high counter, after which the real peer's in-window packets are rejected as "behind window" — a remote DoS that needs no key. Move the replay check after a successful Open so only authenticated counters move the window (the same auth-then-replay order WireGuard uses). Tradeoff: a replayed frame is now decrypted before being rejected, which is the accepted industry choice. APO-647 / S4 — drop an empty decrypted payload before reading its version nibble. The min-size guards read ipPacket[0] first and only then check < IPvNMinimumSize, so they catch 1..19 bytes but still panic on exactly 0. An authenticated peer can send a non-OOB frame with an empty plaintext (valid GCM: empty body + tag). The encap path already had the len==0 guard; this adds the symmetric one to decap. The differential fuzzer can't reach this (it needs a valid tag), which is why it slipped the earlier crash-hardening pass.
The processFrames loop holds runtime.LockOSThread and has no recovery, so a single panicking packet would tear down the whole queue goroutine (and, via errgroup, the forwarder). The transforms are written to drop malformed frames rather than panic, so this is a last-resort backstop: it converts any panic into a frame drop and keeps the queue running. It also contains the GCM inexact-overlap panic class the in-place aliasing contract depends on. The recovered-panic log is bounded to a single emission so a crafted frame cannot flood the logs.
First piece of the key-establishment control plane: the PSP-model key derivation that turns an authenticated, forward-secret session into per-Security-Association AEAD keys for the existing Geneve/AF_XDP data plane. - cmac.go: AES-CMAC (NIST SP 800-38B / RFC 4493) built on crypto/aes so the derivation stays inside the Go FIPS 140-3 module. - kdf.go: the SP 800-108 KDF and PSPVersion codepoints selecting AEAD (AES-GCM-128/256) and derived key size. - kdf_test.go: test vectors.
Establish authenticated, forward-secret per-session keys over a dedicated QUIC/mTLS control channel, following the PSP security model while keeping the Geneve/AF_XDP data plane untouched. Resolves the static-PSK / no-forward-secrecy gap (APO-648/S5); the SPI-bound nonce that closes the GCM reuse window (S1) lands with the Phase 3 data-plane wiring. - identity.go: long-term ECDSA P-256 identities, WireGuard-style pinned raw public keys (self-signed leaf, SPKI pin), genkey/pubkey-ready marshalling. - tls.go: TLS 1.3-only config, ALPN icx-ctrl/1, pin verifier via VerifyConnection, RFC 8446 exporter -> 32-byte root secret. Under fips140=on the suite is AES-GCM + P-256/384 + SHA-2 + HKDF with no custom handshake crypto. - transport.go: QUIC Dial/Listen (Retry-based address validation is the handshake-flood defense), no 0-RTT so every handshake is a fresh ECDHE (forward secrecy). Root secret -> HKDF master keys; SA negotiation over a QUIC stream where each peer announces only its RX SPI and both derive every key locally -- no key material on the wire. A successful NegotiateSAs, not a successful Dial, is the fail-closed key-confirmation precondition. - sa.go/protocol.go: SPI allocation (MSB master-key selector, role-partitioned so tx key != rx key), SP800-108/AES-CMAC SA derivation, framed SA-offer. - kdf.go: PSP version now fails closed on unknown codepoints instead of defaulting to a 16-byte key. - _examples/keyexchange: runnable, self-verifying two-peer loopback demo that tracks the API; doubles as a smoke test (GODEBUG=fips140=on go run ...). Adds quic-go v0.59.1 (no crypto of its own; delegates to crypto/tls, so the FIPS boundary stays the Go module).
… seam (APO-644) Reshape the data plane toward the PSP nonce model and harden the key-install path, keeping the Geneve/AF_XDP wire format unchanged. Nonce = epoch‖counter. The 12-byte GCM nonce was 0x00000000‖counter64; the high 4 bytes are now the 32-bit SPI (the value already carried in the Geneve key-epoch option). All four TX sites (VirtToPhy, ToPhy, and the in-place equivalents) write it before Seal; both decap sites verify nonce[:4] == the selected epoch before Open and drop + count RXDropsSPIMismatch on a mismatch. GCM already authenticates these bytes (nonce + header are in the tag), so the check is fast-path tamper-rejection and a precise drop reason rather than new authenticity. The decap parsers now also require the TxCounter option's declared 12-byte length before trusting its value as the nonce. Key-install seam. UpdateVirtualNetworkKeys is split into a guarded public seam over an unguarded installKeys. The seam fails closed on three invariants: a non-zero epoch (matching control/sa.go's reserved-SPI rule), a strictly increasing epoch, and rxKey != txKey. The last is load-bearing: both simplex directions share the single epoch, so the SPI prefix is identical inbound and outbound and the keys are the only thing separating the two directions' nonce spaces — equal keys would collide nonces under one key. Scope / honesty: - This closes in-process re-install/rotation reuse and adds RX SPI tamper rejection. The cross-restart static-key case (counter resets to 0 under a reused key) is NOT closed here: the monotonicity guard compares in-memory state only. Restart safety comes from ephemeral per-session keys (the control plane) or durable epoch/counter state — Phase 4. The doc on UpdateVirtualNetworkKeys states this and the per-VNI serialization precondition. - A single shared epoch cannot represent the control plane's distinct tx/rx SPIs; a TODO marks where Phase 4 wires them in. - Hard flag day: a Phase-3 receiver drops frames from a pre-Phase-3 sender (nonce[:4]=0 != epoch). Peers must upgrade together per VNI; preserving the old all-zero-prefix format would reopen the reuse hazard. Tests. The single-handler loopback tests (byte-equivalence, round-trip, fuzz, bench) use one shared key, which the distinct-key guard now forbids, so they route through a test-only InstallKeysForTest seam (export_test.go, compiled only under `go test`). The foreign-package forwarder crypto test is split into two peer handlers with distinct keys. Adds TestUpdateVirtualNetworkKeysGuards and TestRXRejectsSPINonceMismatch. The in-place vs cross-buffer fuzz oracle still agrees in both directions after the nonce binding. Reviewed adversarially (22-agent workflow): no must-fix; the confirmed findings (restart-reuse scoping, shared-epoch phrasing, flag-day, reserved-SPI, option length) are folded in as guards and precise comments above.
Reflect the key-install guards added in f7332ab: rx and tx must differ and the epoch must strictly increase within a process, so the CLI now refuses an INI whose rx==tx. Spell out that a restart re-reads the INI at epoch 1 with the TX counter reset to 0, so restarting against an unchanged key file reuses the AES-GCM nonce sequence — rotate to fresh keys or use a key-exchange mechanism.
…atapath (APO-648) Drive the QUIC/mTLS control plane into the data plane. A Tunnel establishes the session, negotiates the first SA pair fail-closed, and keeps keys fresh: the initiator rekeys on a timer (reacting to QUIC connection-context loss), the responder serves rekeys from a blocking accept loop. Any negotiation error is session-fatal and triggers reconnect-with-backoff (no retry on a dead session); a rejected install (epoch regression after a reconnect) is logged and swallowed so the data plane keeps its current keys and fails closed on their expiry. Bridge the control plane's distinct, role-partitioned per-direction SPIs onto the handler's single shared epoch via SharedEpoch (the initiator-allocated / bit30==0 SPI): both peers install the identical scalar epoch while deriving distinct per-direction keys from the distinct SPIs, so AES-GCM nonce uniqueness rests on the distinct-key guard. This needs no handler/seam change; carrying the genuine per-direction SPI on the wire is a future additive UpdateVirtualNetworkSAs. Add SelectMode (static XOR control-plane, fail-closed, no silent fallback), CanonicalInitiator (deterministic role election by SPKI order, equal keys rejected), and Session.Context for prompt session-loss detection. Tests: SelectMode truth table; CanonicalInitiator ordering + equal-key reject; SharedEpoch agreement/validation; installSAs PSPv0/16-byte fail-closed and swallow-on-rejection; two-peer loopback bring-up, rekey, pin-mismatch fail-closed, and reconnect self-heal; plus an end-to-end control-plane->handler Geneve round-trip proving SharedEpoch interoperates with zero drops while the naive per-direction Tx.SPI epoch drops every frame.
…644) Add a control-plane keying mode alongside the legacy static INI keys, selected fail-closed: exactly one of --key-file or --identity-key/--peer-key, with no silent fallback between them. In control-plane mode the CLI loads the identity, pins the peer key, binds a dedicated control UDP socket, brings the tunnel up synchronously (the forwarder never starts without keys), and runs the rekey loop and forwarder under one errgroup sharing a cancel. SIGHUP static reload is gated to static mode so exactly one installer touches a VNI. This closes the production APO-644 path: ephemeral, forward-secret per-session keys mean a restart yields fresh keys, so the static-INI restart nonce-reuse hazard does not apply in control-plane mode. New commands: icx genkey (writes a P-256 identity, O_CREATE|O_EXCL 0600, --force to overwrite) and icx pubkey. New flags: --identity-key, --peer-key, --control-port, --peer-control-port, --rekey-interval, --require-fips. Reject --control-port == --port (the XDP filter would blackhole the control plane). Use SIGINT/SIGTERM for graceful shutdown. README documents the control plane as the recommended path, marks static keys legacy, and records the startup-ordering and one-sided-restart caveats.
Durable-epoch seeding makes the data-plane epoch climb monotonically across rekeys and restarts; AES-GCM nonce uniqueness then rests on each epoch install starting its own TX counter at zero (nonce = epoch‖counter). That reset already happens (installKeys stores a fresh transmitCipher), but nothing pinned it, so a future refactor that carried the counter across installs would silently reintroduce (key, nonce) reuse. Document the invariant at the install site and add a white-box test (TxCounterForTest seam + TestInstallResetsTxCounterPerEpoch) that installs two epochs and asserts the counter resets to zero and counts from 1 again.
… (APO-648)
The per-session SPI allocator resets on every (re)connect, so the shared
data-plane epoch (the initiator-allocated SPI counter) regressed to 1 and the
survivor's strictly-increasing epoch guard rejected it — breaking a transient
reconnect, a responder restart, and an initiator restart alike.
Carry an epoch high-water forward and seed each new session's allocator above
it. Two layers, both initiator-only (only the initiator's SPI becomes the wire
epoch):
- In memory, always on: fixes a transient reconnect (a latent rekey-deadlock)
and a responder restart for free.
- Durably, opt-in via an EpochStore: a FileEpochStore persists the high-water
with fsync + atomic-rename + dir-fsync, integrity-protected by an
HMAC-SHA256 keyed from the identity's fixed-width private scalar and bound to
the (local, peer) identity pin. Adds initiator-restart recovery; RequireState
fails closed on a corrupt/unreadable/persistently-failing store.
The margin is applied at seed time so it covers both the durable persistence gap
(<=2 generations) and a torn-exchange reconnect lead (1 generation, where the
responder committed an epoch the initiator never recorded). Persistence runs on
a dedicated coalescing goroutine so a wedged fsync cannot freeze the rekey/
reconnect loop; stop() is grace-bounded for the same reason. SPI-space
exhaustion and a stalled store (under RequireState) are terminal: Run fails
closed instead of hot-looping.
SeedFloor/ErrSPIExhausted on the allocator, Session.SeedRxFloor, and the Tunnel
wiring (load at Bringup, seed in establish, persist on install, fatal paths).
Tests cover the store format/corruption/round-trip, the persister, the
allocator seed, reconnect monotonicity under an enforcing guard, the
torn-exchange lead, the durable restart round-trip, and the responder ignoring
its store.
Expose the control plane's durable epoch high-water: --state-file enables a FileEpochStore (built role-agnostically; the Tunnel consults it only when this node is the elected initiator, so set it on both peers), and --require-state fails closed on a corrupt/unreadable/un-writable store instead of degrading to a fresh start. --require-state requires --state-file. Rewrite the README "Restart / reconnect" notes to describe the two recovery layers (in-memory always-on; durable with --state-file), the both-peers requirement, and the integrity-tripwire scope (the MAC is not rollback- or deletion-resistant, which is acceptable because per-session ephemeral keys mean a rolled-back high-water cannot cause nonce reuse — only a transient outage).
Disable TLS 1.3 session resumption (SessionTicketsDisabled) and fail closed in newSession on a resumed session or 0-RTT (DidResume / Used0RTT). The data plane's AES-GCM nonce uniqueness rests on every control session deriving fresh master keys, so that a per-direction SPI which resets or regresses after a reconnect is always paired with a never-used key. Forbidding resumption is the foundation that lets the install seam accept a reset SPI without persisted state.
…t guard (APO-644) UpdateVirtualNetworkSAs installs the two PSP simplex SAs under their own role-partitioned SPIs (rxSPI we decrypt under, txSPI we encrypt to), replacing the single shared epoch. The AES-GCM nonce is txSPI‖counter with the counter reset to zero per generation, so nonce uniqueness across rekeys, reconnects and restarts rests on each generation pairing that from-zero counter with a fresh per-session key. Because the control plane now guarantees fresh keys per session, this path does NOT enforce SPI monotonicity: a reconnect resets the allocator to a low SPI and that reset SPI must be re-accepted under its fresh key. The only fail-closed guards are non-zero SPIs, distinct rx/tx keys, and a TX anti-reset check that rejects re-installing the CURRENTLY-live transmit SA — same SPI AND same key. The key comparison (new transmitCipher.key) is load-bearing: on a transient reconnect the allocator can hand back a transmit SPI that collides with the still-live one, but under a fresh key, which is safe; comparing the SPI alone would spuriously reject that recovery. The legacy static-key seam (UpdateVirtualNetworkKeys) keeps STRICT epoch monotonicity, since static pre-shared keys carry no per-session freshness.
… (APO-648) The Tunnel now hands the handler the two genuine per-direction SPIs from NegotiateSAs (rxSPI = our receive SPI, txSPI = the peer's) via the reshaped SAInstaller, instead of collapsing them onto one SharedEpoch. With fresh master keys per session (see the fresh-ECDHE enforcement) a reset/regressed SPI is always paired with a fresh key, so no persisted epoch state is needed for recovery: the handler accepts the reset SPI and a transient reconnect or a one-sided restart of either peer recovers seamlessly with zero on-disk state. Removes the whole receive-SPI high-water layer that the old strict monotonicity guard forced: SharedEpoch/roleBit, the EpochStore/persister and its fatal tripwire, SeedFloor/SeedRxFloor, and the in-memory/durable high-water tracking (control/epochstate.go and its test). Reconnect recovery is now proven by TestTunnelReconnects / TestTunnelReconnectGuardNeverRejects, whose guardInstaller mirrors the handler's same-SPI-AND-same-key anti-reset guard.
…-648) The control-plane installer now forwards both per-direction SPIs to UpdateVirtualNetworkSAs. Removes the --state-file/--require-state flags and the durable-epoch wiring (EpochStore/RequireState on the Tunnel): with fresh per-session keys there is no high-water to persist, so a restart recovers with no on-disk state.
…(APO-648) Remove the --state-file/--require-state flag documentation and rewrite the "Restart / reconnect" section: recovery is now seamless and symmetric with no persisted state to manage, because every (re)connect is a fresh ECDHE handshake whose fresh keys make a reset per-direction SPI safe.
Remove the legacy static-key mode: the --key-file flag, runStaticKeying, the SIGHUP INI reload loop, loadKeysFromINI, and the gopkg.in/ini.v1 dependency. The QUIC/mTLS control plane is now the only keying path — run requires both --identity-key and --peer-key, fail-closed, and always drives the rekey loop under the errgroup. This closes the last AES-GCM nonce-reuse path (the residual APO-644 hazard): the static path re-read the INI at epoch 1 on restart and reset the TX counter under an unchanged persisted key. With the control plane as the sole installer, every session derives fresh per-session ECDHE keys, so a reset counter is never paired with a reused key.
…seam With static-INI keying gone, UpdateVirtualNetworkKeys has no production caller and is no longer "the static --key-file path". Re-document it as the simple single-epoch manual- keying seam used by tests and embedders (the control plane installs genuine per-direction SAs via UpdateVirtualNetworkSAs). The strict epoch-monotonicity and distinct-key guards are unchanged; the doc now states the caller's cross-restart responsibility for a manually-supplied key that survives restarts.
The control plane is the only keying mode now, so the Mode enum, its String method, and SelectMode (which chose between static-INI and control-plane keying) are dead. Remove them along with TestSelectMode; the CLI validates the required identity/peer keys directly.
…-758)
PSPVersion / PSPv0 / PSPv1 named a PSP wire attribute icx does not emit — icx
borrows PSP's key schedule but runs no PSP packet format (the data plane is
Geneve + AF_XDP). Rename the local AEAD cipher-suite selector to ICXVersion, with
AESGCM128 / AESGCM256 for the two suites, and the saOffer field to Version
(matching SA.Version). The genuine PSP key-schedule primitives (DeriveSAKey,
MasterKeys, MasterKeyIndex, the SPI bit-layout, the CMAC PRF) keep the PSP name —
they implement the spec and the name is a correct pointer to it.
Pure Go-identifier rename, no byte changes: the on-wire codepoints (0/1), the KDF
label bytes ("Pv0\0"/"Pv1\0"), and the masterKeyInfo domain string are untouched,
and the PSP-spec KDF vector test (TestDeriveSAKey_PSPSpec) still passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
ICX previously loaded static 16-byte PSKs from a plaintext INI and fed them straight into AES-128-GCM with an AEAD nonce of
0x00000000‖counter64and a process-local counter that reset to 0 on restart. Under a long-lived static key that is catastrophic GCM nonce reuse (S1 / APO-644), and there was no key exchange and no forward secrecy (S5 / APO-648) — one key-file or memory disclosure decrypts all recorded past and future traffic.Both stem from one root cause: a long-lived key with a process-local counter. This PR replaces that with fresh, authenticated, per-session keys established over a QUIC/mTLS control plane, reshaping the key/nonce model to follow the Google PSP architecture while keeping ICX's existing Geneve/AF_XDP data plane on the wire. The whole design stays on FIPS-approved primitives (P-256/384 ECDHE, AES-GCM, SHA-2, HKDF, AES-CMAC/SP 800-108) so it is certifiable under Go 1.24's native FIPS 140-3 module (
GODEBUG=fips140=on).What changed
Control plane (new
control/package)control/cmac.go,control/kdf.go), validated against the PSP spec's published test vectors.control/transport.go,control/protocol.go,control/sa.go).control/cp.go).Data plane (
handler.go, minimal touch)nonce[:4] == SPI), turning the formerly-ignored zero bytes into an integrity check and giving per-SA nonce separation.Forwarder
recover()converts datapath-transform panics into frame drops instead of taking down the busy loop.Migration / cleanup
--key-file,loadKeysFromINI, the SIGHUP reload, and theMode/SelectModemachinery. The control plane is now required, fail-closed.PSPVersion → ICXVersion(PSPv0/PSPv1 → AESGCM128/AESGCM256) since the data plane is not PSP (APO-758); wire bytes and the PSP key-schedule names are unchanged.FIPS posture
Every primitive is in the Go FIPS 140-3 module: TLS 1.3 ECDHE P-256/384 + AES-GCM + SHA-2 + HKDF, QUIC AES-GCM packet protection, HKDF-SHA-256 master-key seed, SP 800-108/AES-CMAC SA derivation over the validated
crypto/aes, and AES-128/256-GCM on the data plane.quic-gocontains no crypto — it delegates tocrypto/tls— so the validation boundary is just the Go module.Testing
control+ root packages:go build,go vet, and fullgo testgreen.GOOS=linux go build ./...green.handler_test.go/cp_wire_test.go.Tickets
Resolves APO-644 (S1 nonce reuse), APO-648 (S5 forward secrecy / control plane), APO-645 (replay-after-auth), APO-647 (empty-payload guard), APO-758 (PSPVersion→ICXVersion rename).
🤖 Generated with Claude Code