diff --git a/docs/user/README.md b/docs/user/README.md index cfaef34fe1..fb004b73ef 100644 --- a/docs/user/README.md +++ b/docs/user/README.md @@ -8,6 +8,8 @@ List of the documents in alphabetical file name order. - [Getting Started with MicroShift](./getting_started.md) - [Integrating MicroShift with Greenboot](./greenboot.md) - [AMQ Broker on MicroShift](./howto_amq_broker.md) +- [Cluster-to-Cluster Connectivity (C2CC)](./howto_c2cc.md) +- [Encrypting C2CC Traffic with IPsec](./howto_c2cc_ipsec.md) - [MicroShift Configuration](./howto_config.md) - [Firewall Configuration](./howto_firewall.md) - [Gateway API](./howto_gateway_api.md) diff --git a/docs/user/howto_c2cc.md b/docs/user/howto_c2cc.md new file mode 100644 index 0000000000..e03663db2d --- /dev/null +++ b/docs/user/howto_c2cc.md @@ -0,0 +1,459 @@ +# Cluster-to-Cluster Connectivity (C2CC) + +Cluster-to-Cluster Connectivity (C2CC) enables direct Pod-to-Pod and +Pod-to-Service communication between independent MicroShift clusters. +It targets edge deployments where multiple single-node MicroShift instances +on the same network segment (or reachable via routable next-hops) need to +consume each other's workloads without an external interconnect solution. + +C2CC provides: + +- **Pod-to-Pod communication** across clusters using pod IPs. +- **Pod-to-Service communication** across clusters (ClusterIP and headless + Services) using DNS names such as + `myservice.mynamespace.svc.cluster-b.remote`. +- **Source pod IP preservation** — cross-cluster traffic bypasses SNAT, so + NetworkPolicies on the remote cluster can enforce access control based on + the originating pod IP. +- **Health monitoring** — per-remote-cluster health and latency reporting + through the `RemoteCluster` custom resource. +- IPv4, IPv6, and dual-stack support. + +C2CC is built into MicroShift and driven entirely by the MicroShift +configuration file. +There is no separate service to install: adding entries to +`clusterToCluster.remoteClusters` enables the feature, removing them disables +it and cleans up all C2CC-owned state. + +Cross-cluster traffic travels as plain routed IP over the underlay network — +there is no tunnel between the hosts and no encryption by default. +For production deployments, encrypting and authenticating this traffic with +IPsec is strongly recommended. +See [Encrypting C2CC Traffic with IPsec](./howto_c2cc_ipsec.md). + +## How It Works + +Once enabled, the C2CC controller inside MicroShift automatically manages +several routing and NAT subsystems on the host: + +- **OVN static routes** — routes on the OVN gateway router (`GR_`) + send traffic destined to remote pod and service CIDRs to the configured + next-hop instead of the default gateway. C2CC-owned routes are tagged with + `external_ids:k8s.ovn.org/owner-controller=microshift-c2cc` (visible via + `ovn-nbctl lr-route-list GR_`). +- **SNAT bypass** — nftables rules (commented `c2cc-no-masq`) in + OVN-Kubernetes' `ovn-kube-pod-subnet-masq` chain and the node annotation + `k8s.ovn.org/node-ingress-snat-exclude-subnets` prevent masquerading of + cross-cluster traffic, preserving original pod source IPs end-to-end. +- **Linux policy routing** — routes to remote CIDRs live in a dedicated + routing table (default 200, see `ip route show table 200`). A second table + (default 201) reroutes inbound remote-to-local-service traffic through the + OVN management port so that service load balancing does not SNAT the + source. +- **CoreDNS forwarding** — for each remote cluster with a `domain` + configured, a CoreDNS server block rewrites the remote domain to + `cluster.local` and forwards the query to the remote cluster's DNS + service. +- **Health probes** — a lightweight `c2cc-probe` Deployment in the + `openshift-c2cc` namespace probes each remote cluster's probe Service + through the full C2CC data path and reports the results in + `RemoteCluster` custom resources. + +The controller reconciles continuously (event-driven, with a periodic +fallback every few seconds). +If routes, rules, or annotations are removed — for example by an +OVN-Kubernetes restart, a firewall reload, or a host reboot — they are +restored automatically within seconds. + +## Prerequisites + +- Two or more hosts running MicroShift with the default OVN-Kubernetes CNI + (C2CC requires OVN-Kubernetes; it is not available with + `network.cniPlugin: none`). +- IP connectivity between the hosts on the underlay network (same L2 + segment, or routable next-hops). +- **Non-overlapping pod and service CIDRs across all clusters.** + MicroShift's defaults are `10.42.0.0/16` (pods) and `10.43.0.0/16` + (services), so all clusters except one must override them. Plan the CIDRs + up front: changing `network.clusterNetwork` or `network.serviceNetwork` on + a cluster that has already started requires wiping MicroShift data + (`microshift-cleanup-data --all`), which deletes all workloads. +- **Bidirectional configuration.** Both clusters must configure each other + as remotes. One-way setups are not supported. + +## Configure C2CC + +The examples below assume a two-cluster setup: + +| Host | Underlay IP | Pod CIDR | Service CIDR | Domain | +|---------|---------------|----------------|----------------|--------------------| +| Host A | 192.168.1.10 | 10.42.0.0/16 | 10.43.0.0/16 | cluster-a.remote | +| Host B | 192.168.1.20 | 10.45.0.0/16 | 10.46.0.0/16 | cluster-b.remote | + +### Override cluster and service networks + +Host A uses the MicroShift defaults. +On **Host B**, override the pod and service networks before MicroShift +starts for the first time, for example in `/etc/microshift/config.yaml`: + +```yaml +network: + clusterNetwork: + - 10.45.0.0/16 + serviceNetwork: + - 10.46.0.0/16 +``` + +### Add the remote cluster configuration + +On each host, create a configuration drop-in pointing at the remote cluster. + +On **Host A**, create `/etc/microshift/config.d/50-c2cc.yaml`: + +```yaml +clusterToCluster: + remoteClusters: + - nextHop: + - 192.168.1.20 + clusterNetwork: + - 10.45.0.0/16 + serviceNetwork: + - 10.46.0.0/16 + domain: cluster-b.remote +``` + +On **Host B**, create the same file describing Host A: + +```yaml +clusterToCluster: + remoteClusters: + - nextHop: + - 192.168.1.10 + clusterNetwork: + - 10.42.0.0/16 + serviceNetwork: + - 10.43.0.0/16 + domain: cluster-a.remote +``` + +For each remote cluster: + +- **`nextHop`** — IP addresses of the remote cluster's node, used as the + routing next-hop. At most one IPv4 and one IPv6 address; dual-stack + clusters need both. +- **`clusterNetwork` / `serviceNetwork`** — the remote cluster's pod and + service CIDRs. They must not overlap with the local networks or with any + other remote. +- **`domain`** — optional DNS suffix for the remote cluster. When set, + Services on the remote cluster are resolvable as `..svc.`. + When empty, no DNS forwarding is configured for this remote. + +To connect more than two clusters, add one `remoteClusters` entry per remote +on every host (full mesh, N×(N-1) entries in total). + +### Configure the firewall + +MicroShift intentionally does not manage firewall rules for C2CC — edge +deployments often have site-specific firewall policies. +On each host, add the *remote* cluster's pod and service CIDRs to the +trusted zone: + +```bash +# On Host A — trust Host B's pod and service CIDRs +sudo firewall-cmd --permanent --zone=trusted --add-source=10.45.0.0/16 +sudo firewall-cmd --permanent --zone=trusted --add-source=10.46.0.0/16 +sudo firewall-cmd --reload +``` + +```bash +# On Host B — trust Host A's pod and service CIDRs +sudo firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 +sudo firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 +sudo firewall-cmd --reload +``` + +Do **not** add the remote host's own IP to the trusted zone. +C2CC is designed for pod-to-pod and pod-to-service traffic only; leaving the +host IP untrusted blocks host-originated traffic from reaching local pods. +(The IPsec setup requires opening UDP 500/4500 and ESP separately — see the +[IPsec guide](./howto_c2cc_ipsec.md).) + +### Restart MicroShift + +```bash +sudo systemctl restart microshift +``` + +MicroShift validates the C2CC configuration on startup. +If validation fails (overlapping CIDRs, a routing loop, an invalid next-hop, +and so on), MicroShift logs the errors and does not start. +All C2CC configuration changes — including DNS cache settings — require a +MicroShift restart to take effect. + +## Verify + +Check that C2CC is active in the journal: + +```bash +journalctl -u microshift | grep C2CC +# ... "C2CC is enabled with 1 remote cluster(s)" +``` + +Check the health of each remote cluster. +Each remote gets a cluster-scoped `RemoteCluster` resource named after its +next-hop IP: + +```bash +$ oc get remoteclusters.microshift.io +NAME AGE +c2cc-192-168-1-20 5m + +$ oc get remoteclusters.microshift.io c2cc-192-168-1-20 -o yaml +apiVersion: microshift.io/v1alpha1 +kind: RemoteCluster +metadata: + name: c2cc-192-168-1-20 + labels: + app.kubernetes.io/managed-by: c2cc-route-manager +spec: + probeInterval: 10s + probeTargets: + - 10.46.0.11:8080 +status: + state: Healthy + lastProbeTime: "2026-07-07T12:12:09Z" + lastSuccessfulProbe: "2026-07-07T12:12:09Z" + targetResults: + - target: 10.46.0.11:8080 + state: Healthy + latency: + avg: 1.579596ms + last: 1.21661ms + max: 3.621629ms + min: 959.724µs + stddev: 512.887µs +``` + +`status.state` transitions from `NeverProbed` to `Healthy` once end-to-end +probes succeed. +`status.targetResults` contains per-target latency statistics measured +through the full C2CC data path, and `status.errors` explains failures. + +Inspect the routes C2CC manages (on Host A, with Host B configured as the +remote): + +```bash +$ ip route show table 200 +10.45.0.0/16 via 192.168.1.20 dev enp1s0 proto 200 +10.46.0.0/16 via 192.168.1.20 dev enp1s0 proto 200 + +$ ip route show table 201 +10.43.0.0/16 via 10.42.0.1 dev ovn-k8s-mp0 proto 201 + +$ ip rule +0: from all lookup local +30: from all fwmark 0x1745ec lookup 7 +99: from 10.45.0.0/16 to 10.43.0.0/16 lookup 201 +99: from 10.46.0.0/16 to 10.43.0.0/16 lookup 201 +100: to 10.45.0.0/16 lookup 200 +100: to 10.46.0.0/16 lookup 200 +32766: from all lookup main +32767: from all lookup default +``` + +The SNAT bypass state is visible in the OVN-Kubernetes nftables chain and +on the Node object: + +```bash +$ sudo nft list chain inet ovn-kubernetes ovn-kube-pod-subnet-masq +table inet ovn-kubernetes { + chain ovn-kube-pod-subnet-masq { + ip daddr 10.45.0.0/16 return comment "c2cc-no-masq:10.45.0.0/16" + ip daddr 10.46.0.0/16 return comment "c2cc-no-masq:10.46.0.0/16" + ip saddr 10.42.0.0/24 masquerade + } +} + +$ oc get node -o jsonpath='{.items[0].metadata.annotations.k8s\.ovn\.org/node-ingress-snat-exclude-subnets}' +["10.45.0.0/16", "10.46.0.0/16"] +``` + +The OVN static routes can be listed through the `ovnkube-master` pod; each +C2CC-owned route carries the `microshift-c2cc` owner tag: + +```bash +$ oc exec -n openshift-ovn-kubernetes daemonset/ovnkube-master -- \ + ovn-nbctl find Logical_Router_Static_Route \ + external_ids:k8s.ovn.org/owner-controller=microshift-c2cc +_uuid : d6708885-c1ac-4001-b49e-d70a3652b0d0 +external_ids : {"k8s.ovn.org/owner-controller"=microshift-c2cc} +ip_prefix : "10.45.0.0/16" +nexthop : "192.168.1.20" +... + +_uuid : 5fd91bce-ab5d-483a-a8b9-50465dad4a96 +external_ids : {"k8s.ovn.org/owner-controller"=microshift-c2cc} +ip_prefix : "10.46.0.0/16" +nexthop : "192.168.1.20" +... +``` + +Finally, test connectivity from a pod — for example, resolve and reach a +Service on the remote cluster: + +```bash +oc exec -- curl -s http://myservice.mynamespace.svc.cluster-b.remote:8080 +``` + +## Cross-Cluster DNS + +When a remote cluster has a `domain` configured, MicroShift's CoreDNS +forwards queries for that domain to the remote cluster's DNS service. +Any Service on the remote cluster is resolvable as: + +```text +..svc. +``` + +Both ClusterIP and headless Services work: ClusterIP Services resolve to the +remote service IP, headless Services resolve directly to remote pod IPs. + +The forwarding is implemented as a per-remote server block in the CoreDNS +Corefile, which you can inspect in the `dns-default` ConfigMap: + +```bash +$ oc get configmap -n openshift-dns dns-default -o jsonpath='{.data.Corefile}' +cluster-b.remote:5353 { + bufsize 1232 + errors + log . { + class error + } + rewrite stop name suffix .cluster-b.remote .cluster.local answer auto + forward . 10.46.0.10 + cache 10 { + denial 9984 10 + } +} +.:5353 { + ... +``` + +DNS responses are cached locally. +The cache TTLs are tunable via `clusterToCluster.dns.cacheTTL` (positive +answers) and `clusterToCluster.dns.cacheNegativeTTL` (NXDOMAIN/NODATA), both +defaulting to 10 seconds. +Lower them if remote service endpoints change rapidly; raise them to reduce +DNS load on the remote cluster. + +## Configuration Reference + +All settings live under the `clusterToCluster` section: + +| Option | Default | Description | +|--------|---------|-------------| +| `remoteClusters` | `[]` | List of remote clusters. C2CC is disabled when empty. | +| `remoteClusters[].nextHop` | — | Next-hop IPs of the remote node. At most one IPv4 and one IPv6 address. Must not equal the local node IP or duplicate another remote. | +| `remoteClusters[].clusterNetwork` | — | Remote pod CIDRs. Must not overlap with local or other remote CIDRs. Minimum mask /8 (IPv4) or /32 (IPv6). | +| `remoteClusters[].serviceNetwork` | — | Remote service CIDRs. Same constraints as `clusterNetwork`; must have the same number of entries with matching IP families. | +| `remoteClusters[].domain` | `""` | Optional DNS suffix for the remote cluster. Must be a valid DNS subdomain, unique across remotes, and not `cluster.local`. | +| `probeInterval` | `10s` | Interval between health probes to each remote. Go duration string between `1s` and `5m`. | +| `dns.cacheTTL` | `10` | Max TTL (seconds) for positive DNS cache entries for remote domains. `0` disables caching. | +| `dns.cacheNegativeTTL` | `10` | Max TTL (seconds) for negative (NXDOMAIN/NODATA) DNS cache entries. `0` disables. | +| `routing.routeTableID` | `200` | Linux policy routing table for routes to remote CIDRs. Range 1–252; must differ from `serviceRouteTableID`. | +| `routing.serviceRouteTableID` | `201` | Linux policy routing table for service routes via the OVN management port. Range 1–252; must differ from `routeTableID`. | + +The routing table IDs only need to be changed if other software on the host +(for example, NetworkManager dispatcher scripts or VPN tooling) already uses +tables 200/201. + +## IPv6 and Dual-Stack + +C2CC supports IPv6-only and dual-stack clusters. +The local and remote clusters must have compatible IP families: for every +remote CIDR family there must be a matching-family `nextHop` entry. + +For a dual-stack remote, list both families: + +```yaml +clusterToCluster: + remoteClusters: + - nextHop: + - 192.168.1.20 + - 2001:db8:0:1::20 + clusterNetwork: + - 10.45.0.0/16 + - fd01:0:0:1::/64 + serviceNetwork: + - 10.46.0.0/16 + - fd02:0:0:1::/112 + domain: cluster-b.remote +``` + +`clusterNetwork` and `serviceNetwork` must have the same number of entries +with matching IP families at each position (at most one IPv4 and one IPv6 +CIDR each). + +## Disabling C2CC + +Remove the C2CC drop-in (or the `remoteClusters` entries) and restart +MicroShift: + +```bash +sudo rm /etc/microshift/config.d/50-c2cc.yaml +sudo systemctl restart microshift +``` + +On startup, the controller detects that C2CC is disabled and cleans up all +C2CC-owned state: OVN static routes, kernel routes and rules, nftables +rules, node annotations, the probe deployment, and the `RemoteCluster` +resources. +The journal logs `C2CC is disabled - attempting best effort cleanup`. + +If you overrode `routing.routeTableID` or `routing.serviceRouteTableID`, +disable in two stages so the cleanup targets the right tables: first remove +only the `remoteClusters` entries while keeping the `routing` overrides and +restart, then remove the rest of the `clusterToCluster` section and restart +again. + +Remember to remove the remote CIDRs from the firewall trusted zone as well. + +## Considerations + +- **No authentication or encryption by default.** Any host that can reach + (or spoof) the configured next-hop IP can inject traffic into the + cluster, and because SNAT is bypassed, that traffic can carry an + attacker-chosen source IP that matches NetworkPolicy allow rules. Use + [IPsec](./howto_c2cc_ipsec.md) together with the nftables enforcement + described there for production deployments — that combination provides + encryption and mutual authentication, and drops unauthenticated traffic + (including host-originated traffic to pods) at the network layer even + when the IPsec service itself is stopped or misconfigured. +- **NetworkPolicies are your responsibility.** C2CC does not create + NetworkPolicy resources. Namespaces with default-deny ingress must + explicitly allow the remote pod CIDRs (`ipBlock` selectors work, since + original pod source IPs are preserved). +- **Restart required for changes.** The C2CC configuration is read at + startup; any change (adding/removing remotes, DNS TTLs, routing tables) + requires a MicroShift restart. +- **Changing routing table IDs leaves stale state.** MicroShift does not + clean up C2CC routes on shutdown (by design, so workloads keep network + connectivity if the MicroShift process stops). If you change `routing.*` + table IDs while C2CC is enabled, the routes and rules in the old tables + remain until a reboot or manual cleanup (`ip route flush table `). +- **MTU.** C2CC does not adjust MTU, and cross-cluster traffic itself adds + no encapsulation — it leaves the host as plain IP at the pod MTU. + MicroShift defaults the pod MTU to the MTU of the physical interface; + on jumbo-frame networks it can be set explicitly by creating + `/etc/microshift/ovn.yaml` with an `mtu:` value (changing it requires a + node reboot to take effect). When encrypting the traffic with IPsec, + leave headroom for the ESP overhead when sizing the pod MTU; see the + [IPsec guide](./howto_c2cc_ipsec.md). +- **Scale.** Configuration is static and full-mesh (each cluster lists + every other cluster), so it does not scale to large fleets without + external configuration automation. C2CC is validated with up to 3 + interconnected clusters. +- **Validation failures block startup.** Invalid C2CC configuration + (overlapping CIDRs, routing loops, bad masks, duplicate next-hops or + domains) prevents MicroShift from starting; check + `journalctl -u microshift` for the specific error. diff --git a/docs/user/howto_c2cc_ipsec.md b/docs/user/howto_c2cc_ipsec.md index cdb60accbd..77f5365189 100644 --- a/docs/user/howto_c2cc_ipsec.md +++ b/docs/user/howto_c2cc_ipsec.md @@ -13,6 +13,7 @@ For comprehensive IPsec/VPN documentation, see [Setting up an IPsec VPN](https:/ ## Prerequisites - Two or more RHEL hosts running MicroShift with C2CC configured (non-overlapping pod and service CIDRs, `clusterToCluster.remoteClusters` populated in each node's config). + See [Cluster-to-Cluster Connectivity (C2CC)](./howto_c2cc.md) for the C2CC setup. - IP connectivity between the hosts on the underlay network. - Libreswan installed on every host: @@ -128,27 +129,98 @@ sudo systemctl enable --now ipsec Check that tunnel SAs are established: ```bash -sudo ipsec trafficstatus +$ sudo ipsec trafficstatus +#8: "c2cc-to-host-b/1x1", type=ESP, add_time=1783531540, inBytes=1842, outBytes=1842, maxBytes=2^63B, id='192.168.1.20' +#10: "c2cc-to-host-b/1x2", type=ESP, add_time=1783531540, inBytes=3386, outBytes=3707, maxBytes=2^63B, id='192.168.1.20' +#9: "c2cc-to-host-b/2x1", type=ESP, add_time=1783531540, inBytes=3287, outBytes=3052, maxBytes=2^63B, id='192.168.1.20' +#11: "c2cc-to-host-b/2x2", type=ESP, add_time=1783531540, inBytes=0, outBytes=0, maxBytes=2^63B, id='192.168.1.20' ``` -You should see output containing `type=ESP` entries for each subnet pair. -For a two-cluster setup with 2 local CIDRs and 2 remote CIDRs, expect 4 child SAs. +You should see `type=ESP` entries for each subnet pair — the `/1x1`, `/1x2`, `/2x1`, `/2x2` suffixes are the combinations of the `leftsubnets`/`rightsubnets` selectors. +Each remote produces 4 child SAs (2 local CIDRs × 2 remote CIDRs), so a two-cluster setup shows 4 and a three-cluster full mesh shows 8 per host. +On RHEL 9 (Libreswan 4.x), each line is additionally prefixed with a `006` status code. -Verify XFRM state is populated: +Verify XFRM state is populated and that traffic actually flows through the SAs — the byte counters must increase while cross-cluster traffic is running: ```bash -ip xfrm state +$ sudo ip -s xfrm state +src 192.168.1.10 dst 192.168.1.20 + proto esp spi 0xca56d890(3394689168) reqid 16409(0x00004019) mode tunnel + replay-window 0 seq 0x00000000 flag af-unspec esn (0x10100000) + auth-trunc hmac(sha256) 0x... (256 bits) 128 + enc cbc(aes) 0x... (256 bits) + lastused 2026-07-08 17:25:53 + anti-replay esn context: + seq-hi 0x0, seq 0x0, oseq-hi 0x0, oseq 0xc + replay_window 0, bitmap-length 0 + dir out + lifetime config: + ... + lifetime current: + 832(bytes), 12(packets) + add 2026-07-08 17:25:40 use 2026-07-08 17:25:43 + stats: + replay-window 0 replay 0 failed 0 +src 192.168.1.20 dst 192.168.1.10 + proto esp spi 0xa1a985b4(2712241588) reqid 16409(0x00004019) mode tunnel + ... ``` -You can also capture packets on the wire to confirm ESP encapsulation: +There is one state entry per direction per child SA; the keys are shown in full in the real output (truncated to `0x...` here). +To watch the total encrypted byte count as a single number (useful for before/after comparison around a test transfer): ```bash -sudo tcpdump -i enp1s0 -c 10 esp +$ sudo ip -s xfrm state | awk '/bytes/{gsub(/[^0-9]/,"",$1); sum+=$1} END{print sum+0}' +6231 ``` +You can also capture packets on the wire to confirm ESP encapsulation — cross-cluster traffic must appear as ESP, never as plaintext TCP/UDP between pod IPs: + +```bash +$ sudo tcpdump -ni enp1s0 -c 10 esp +17:25:53.766575 IP 192.168.1.10 > 192.168.1.20: ESP(spi=0xca56d890,seq=0x7), length 104 +17:25:53.766989 IP 192.168.1.20 > 192.168.1.10: ESP(spi=0xa1a985b4,seq=0x7), length 104 +... +``` + +## Enforce Encryption with nftables + +The `failureshunt=drop` and `negotiationshunt=drop` options prevent plaintext fallback, but they are enforced by Libreswan itself — they offer no protection if the IPsec service is stopped or a connection definition is removed. +For defense in depth, add kernel-level nftables rules that drop any traffic reaching local pod or service CIDRs without IPsec protection (requires kernel 5.10+ for the `meta ipsec missing` match). +The rules must be attached to the `forward` hook: inbound cross-cluster packets are not addressed to the host itself but forwarded by the host into the OVN network, so an `input`-hook chain would never see them. + +On **Host A**: + +```bash +sudo nft add table inet c2cc_ipsec +sudo nft 'add chain inet c2cc_ipsec enforce { type filter hook forward priority -150; policy accept; }' +sudo nft add rule inet c2cc_ipsec enforce ip daddr 10.42.0.0/16 meta ipsec missing counter drop +sudo nft add rule inet c2cc_ipsec enforce ip daddr 10.43.0.0/16 meta ipsec missing counter drop +``` + +On **Host B**, use its own local CIDRs (`10.45.0.0/16`, `10.46.0.0/16`). + +With these rules in place, inbound cross-cluster traffic is accepted only when it arrived through an IPsec SA — even if IPsec is misconfigured or stopped on the sending side, unencrypted packets are dropped rather than delivered. +The `counter` keyword lets you watch the drop count — it should stay at zero while the tunnels are healthy: + +```bash +$ sudo nft list table inet c2cc_ipsec +table inet c2cc_ipsec { + chain enforce { + type filter hook forward priority -150; policy accept; + ip daddr 10.42.0.0/16 meta ipsec missing counter packets 0 bytes 0 drop + ip daddr 10.43.0.0/16 meta ipsec missing counter packets 0 bytes 0 drop + } +} +``` + +Note that these rules are not persistent across reboots. +To make them permanent, add them to `/etc/sysconfig/nftables.conf` and enable the `nftables` service, or ship them via your image build. + ## Considerations -- **IPsec adds overhead.** ESP tunnel mode adds approximately 36-52 bytes per packet. If you experience MTU issues, verify that path MTU discovery is working or adjust MTU settings accordingly. +- **IPsec adds overhead.** ESP tunnel mode adds roughly 73-93 bytes per packet (ESP header/trailer plus the outer IP header, depending on the cipher suite). Cross-cluster C2CC traffic travels as plain routed IP — it is not Geneve-encapsulated between the hosts — so the largest packet entering the tunnel is the pod MTU, which MicroShift by default sets equal to the MTU of the physical interface. Size the pod MTU so that it plus the ESP overhead fits within the physical MTU. For example, on a jumbo-frame network (physical MTU 9000), set the pod MTU to 8900 by creating `/etc/microshift/ovn.yaml` with `mtu: 8900` — changing this value requires a node reboot to take effect. Also verify that path MTU discovery works between the pods and that near-MTU-sized payloads pass through the tunnel. +- **IPsec also blocks unauthenticated hosts.** Because the tunnel selectors are scoped to pod and service CIDRs and the shunt policies drop unmatched traffic, a host without valid IPsec credentials — including the peer host itself — cannot reach pods on a remote cluster. This closes the host-to-pod path that plain C2CC leaves open at the routing level. - **Tunnel recovery.** If IPsec is restarted on one host, tunnels renegotiate automatically when `auto=start` is set. No MicroShift restart is required. - **Certificates.** This guide uses pre-shared keys for simplicity. For production deployments, consider certificate-based authentication. See the [RHEL VPN documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) for details. - **Policy enforcement.** The example connection definitions include `failureshunt=drop` and `negotiationshunt=drop` to prevent traffic from falling back to plaintext when the tunnel is down or still negotiating. If you remove these options, traffic matching the tunnel selectors will be sent unencrypted whenever the SA is unavailable. diff --git a/test/resources/ipsec.resource b/test/resources/ipsec.resource index 3ff6f66c22..7aa5cbc7e6 100644 --- a/test/resources/ipsec.resource +++ b/test/resources/ipsec.resource @@ -96,10 +96,13 @@ Add NFTables IPsec Enforcement Rules ... Uses 'meta ipsec missing' match (kernel 5.10+) to detect unencrypted packets. ... In tunnel mode, decrypted packets have pod/service CIDRs as dst — plaintext ... packets arriving without ESP are caught by this rule. + ... The chain hooks into forward: inbound cross-cluster packets are not addressed + ... to the host but forwarded into the OVN network (local gateway mode), so an + ... input-hook chain would never see them. [Arguments] ${alias} ${local_pod_cidr} Command On Cluster ${alias} nft add table inet c2cc_ipsec_test Command On Cluster ${alias} - ... nft 'add chain inet c2cc_ipsec_test enforce { type filter hook input priority -150; policy accept; }' + ... nft 'add chain inet c2cc_ipsec_test enforce { type filter hook forward priority -150; policy accept; }' Command On Cluster ${alias} ... nft add rule inet c2cc_ipsec_test enforce ip daddr ${local_pod_cidr} meta ipsec missing counter drop