From d01b89068fa3189b47f68937b274f47e0ee3777e Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Thu, 23 Jul 2026 11:38:11 +0100 Subject: [PATCH 1/6] TELCODOCS-2914 Dedicate CPU resources for DPDK based workloads --- _topic_maps/_topic_map.yml | 3 + modules/cnf-configuring-ovs-dpdk-cpus.adoc | 88 +++++++++++++++++ ...erformance-profile-ovs-dpdk-reference.adoc | 57 +++++++++++ ...-understanding-ovs-dpdk-cpu-isolation.adoc | 67 +++++++++++++ ...rstanding-ovs-dpdk-cpu-load-balancing.adoc | 35 +++++++ .../cnf-verifying-ovs-dpdk-cpu-isolation.adoc | 97 +++++++++++++++++++ .../cnf-reserving-cpus-for-ovs-dpdk.adoc | 32 ++++++ 7 files changed, 379 insertions(+) create mode 100644 modules/cnf-configuring-ovs-dpdk-cpus.adoc create mode 100644 modules/cnf-performance-profile-ovs-dpdk-reference.adoc create mode 100644 modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc create mode 100644 modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc create mode 100644 modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc create mode 100644 scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index e439478b9eb6..bd4f9fd15e6d 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -3517,6 +3517,9 @@ Topics: File: cnf-understanding-low-latency - Name: Tuning nodes for low latency with the performance profile File: cnf-tuning-low-latency-nodes-with-perf-profile +- Name: Reserving CPUs for OVS-DPDK + File: cnf-reserving-cpus-for-ovs-dpdk + Distros: openshift-origin,openshift-enterprise - Name: Tuning hosted control planes for low latency with the performance profile File: cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile - Name: Provisioning real-time and low latency workloads diff --git a/modules/cnf-configuring-ovs-dpdk-cpus.adoc b/modules/cnf-configuring-ovs-dpdk-cpus.adoc new file mode 100644 index 000000000000..e68a5a29174e --- /dev/null +++ b/modules/cnf-configuring-ovs-dpdk-cpus.adoc @@ -0,0 +1,88 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: PROCEDURE +[id="cnf-configuring-ovs-dpdk-cpus_{context}"] += Configuring OVS-DPDK CPUs in a performance profile + +[role="_abstract"] +Create or update a `PerformanceProfile` custom resource (CR) that reserves CPUs for OVS-DPDK by setting `spec.cpu.ovsDpdk`. Optionally disable kernel load balancing on those CPUs with an annotation. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. +* You have installed the OpenShift CLI (`oc`). +* You have identified the CPU topology and NUMA locality of the OVS-DPDK NICs on the target nodes. +* Workload partitioning is enabled, or the kubelet CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. +* You understand that this feature is Technology Preview. + +.Procedure + +. Optional: Inspect CPU and NUMA topology on a target node: ++ +[source,terminal] +---- +$ oc debug node/ -- chroot /host lscpu +---- + +. Plan the CPU budget so that the `reserved`, `ovsDpdk`, and `isolated` sets do not overlap, cover the online CPUs that you intend to use, and leave enough reserved capacity for platform components. On single-node {product-title}, ensure that `reserved` can host both system daemons and control plane workloads. + +. Create a `PerformanceProfile` CR that includes `spec.cpu.ovsDpdk`. ++ +.Example performance profile with OVS-DPDK CPUs +[source,yaml] +---- +apiVersion: performance.openshift.io/v2 +kind: PerformanceProfile +metadata: + name: ovs-dpdk-isolation + annotations: + performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable" <1> +spec: + cpu: + reserved: "0-1" <2> + ovsDpdk: "2-3" <3> + isolated: "4-7" <4> + nodeSelector: + node-role.kubernetes.io/worker: "" +---- +<1> Optional. Set to `disable` to exclude `ovsDpdk` CPUs from kernel load balancing and to configure the OVS-DPDK cgroup partition as `isolated`. Omit the annotation to keep the default `member` partition. +<2> CPUs for operating system and platform components. +<3> CPUs reserved for OVS-DPDK PMD threads. Do not reuse this set for unrelated DPDK applications. +<4> CPUs for Guaranteed application pods. ++ +[IMPORTANT] +==== +The Performance Profile Creator (PPC) tool does not currently generate the `ovsDpdk` field. Create or edit the performance profile YAML manually. +==== + +. Apply the performance profile: ++ +[source,terminal] +---- +$ oc apply -f performance-profile-ovs-dpdk.yaml +---- + +. Wait for the Machine Config Operator to apply the configuration and for the target nodes to reboot. Confirm that the relevant machine config pool has finished updating: ++ +[source,terminal] +---- +$ oc get mcp +---- ++ +The pool that matches your profile selector must show `UPDATED=True` and `UPDATING=False`. + +. Verify that the performance profile is available: ++ +[source,terminal] +---- +$ oc get performanceprofile ovs-dpdk-isolation -o yaml +---- ++ +Confirm that the `Available` condition is `True`. If the profile is `Degraded` with reason `OvsDpdkCPUsPrerequisiteNotMet`, enable workload partitioning or `strict-cpu-reservation` and reapply the profile. + +.Verification + +* Confirm that the `PerformanceProfile` status shows `Available=True`. +* Complete the checks in _Verifying OVS-DPDK CPU isolation_ to confirm node-level configuration. diff --git a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc new file mode 100644 index 000000000000..1654d928e127 --- /dev/null +++ b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc @@ -0,0 +1,57 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: REFERENCE +[id="cnf-performance-profile-ovs-dpdk-reference_{context}"] += Performance profile fields for OVS-DPDK CPUs + +[role="_abstract"] +Use the following reference when you configure OVS-DPDK CPU reservation in a `PerformanceProfile` custom resource (CR). + +== API fields and annotations + +[cols="2,1,3"] +|=== +|Field or annotation |Type |Description + +|`spec.cpu.ovsDpdk` +|string (CPU set) +|Optional. CPUs reserved for OVS-DPDK PMD threads. Example value: `"2-3,8-9"`. Must not overlap `reserved`, `isolated`, or `offlined`. + +|`metadata.annotations["performance.openshift.io/cpu-load-balancing-ovs-dpdk"]` +|string +|Optional. Set to `disable` to exclude `ovsDpdk` CPUs from kernel scheduler load balancing and to set the OVS-DPDK cgroup partition to `isolated`. Omit the annotation or set `enable` to keep the default `member` partition. +|=== + +== Validation and status + +* The `ovsDpdk` CPU set must not overlap the `reserved`, `isolated`, or `offlined` CPU sets. +* Workload partitioning or kubelet `strict-cpu-reservation` is required. If the prerequisite is missing, the profile becomes `Degraded` with reason `OvsDpdkCPUsPrerequisiteNotMet`. +* The Node Tuning Operator adds `ovsDpdk` CPUs to kubelet `reservedSystemCPUs` together with `reserved`, and to TuneD `isolated_cores` together with `isolated`. + +== Example configuration + +[source,yaml] +---- +apiVersion: performance.openshift.io/v2 +kind: PerformanceProfile +metadata: + name: ovs-dpdk-isolation + annotations: + performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable" +spec: + cpu: + reserved: "0-1" + ovsDpdk: "2-3" + isolated: "4-7" + nodeSelector: + node-role.kubernetes.io/worker: "" +---- + +== Limitations + +* Technology Preview support only. +* Specific to OVS-DPDK. Not a generic dedicated-CPU pool for other DPDK applications. +* The Performance Profile Creator (PPC) tool does not generate `spec.cpu.ovsDpdk` in this release. Edit the performance profile manually. +* Full effect of `performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"` depends on matching OVS-side support. diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc new file mode 100644 index 000000000000..212599d4fded --- /dev/null +++ b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc @@ -0,0 +1,67 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: CONCEPT +[id="cnf-understanding-ovs-dpdk-cpu-isolation_{context}"] += Understanding OVS-DPDK CPU isolation + +[role="_abstract"] +OVS-DPDK poll mode driver (PMD) threads run as host processes and continuously poll network interfaces. Any interruption from kernel interrupts, system daemons, or Kubernetes pods can introduce jitter and reduce throughput. Reserving CPUs with `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) gives those threads multi-layer isolation that is comparable to the isolation Guaranteed pods receive on isolated CPUs. + +== Why reserved and isolated CPU sets are not enough + +A performance profile already defines the following CPU sets: + +* `reserved`:: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, Burstable and BestEffort pods can still run on reserved CPUs. +* `isolated`:: CPUs for Guaranteed application pods that the kubelet schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK. + +OVS-DPDK runs outside the pod scheduling path. To give PMD threads exclusive CPUs, the profile needs a third set that is excluded from kubelet scheduling and from host interference sources. + +== The three-tier CPU model for OVS-DPDK + +When you set `spec.cpu.ovsDpdk`, the Node Tuning Operator configures the following CPU roles on matching nodes: + +[cols="1,2,2"] +|=== +|CPU set |Typical use |Isolation behavior + +|`reserved` +|System daemons and platform components +|Excluded from application Guaranteed pods. Workload partitioning or related policies further protect these CPUs. + +|`ovsDpdk` +|OVS-DPDK PMD threads on the host +|Excluded from kubelet scheduling for all QoS classes, banned from `irqbalance`, excluded from systemd CPU affinity, and included in kernel isolation parameters. Placed under the OVS cgroup hierarchy. + +|`isolated` +|Guaranteed latency-sensitive pods +|Available to the kubelet for Guaranteed workloads that request exclusive CPUs. +|=== + +[IMPORTANT] +==== +The `ovsDpdk` CPU set is specific to OVS-DPDK. Do not use it as a generic pool for other DPDK applications. The Node Tuning Operator places these CPUs under the existing `ovs.slice` hierarchy for `ovs-vswitchd`. +==== + +== What the Node Tuning Operator configures + +For CPUs listed in `spec.cpu.ovsDpdk`, the Node Tuning Operator applies configuration across several layers, including the following: + +* Adds the CPUs to the kubelet `reservedSystemCPUs` set together with `spec.cpu.reserved`. +* Adds the CPUs to TuneD `isolated_cores` together with `spec.cpu.isolated`. +* Includes the CPUs in kernel boot parameters such as `isolcpus`, `nohz_full`, and `rcu_nocbs`. +* Excludes the CPUs from `systemd.cpu_affinity`. +* Updates `IRQBALANCE_BANNED_CPUS` so that hardware IRQs are not balanced onto those CPUs. +* Creates and configures `ovsdpdk.slice` under `ovs.slice/ovs-vswitchd.service`, including exclusive CPU assignment for OVS-DPDK. + +== Prerequisites for isolation + +Workload partitioning (`CPUPartitioningAllNodes`) or the kubelet CPU Manager `strict-cpu-reservation` policy option is required. If neither is enabled, the Node Tuning Operator marks the performance profile as `Degraded` with the reason `OvsDpdkCPUsPrerequisiteNotMet` and does not complete the configuration. + +Without one of those prerequisites, Burstable and BestEffort pods can still be scheduled on the OVS-DPDK CPUs and break isolation. + +[NOTE] +==== +You must choose NUMA-aligned CPUs for your OVS-DPDK NICs. The Operator does not automatically assign CPUs based on NIC locality. +==== diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc new file mode 100644 index 000000000000..f5ce8ec1e3ae --- /dev/null +++ b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc @@ -0,0 +1,35 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: CONCEPT +[id="cnf-understanding-ovs-dpdk-cpu-load-balancing_{context}"] += Understanding the OVS-DPDK CPU load-balancing annotation + +[role="_abstract"] +You can optionally exclude `ovsDpdk` CPUs from the kernel scheduler load-balancing pool by setting an annotation on the `PerformanceProfile` custom resource (CR). This control is separate from reserving the CPUs with `spec.cpu.ovsDpdk`. + +== Annotation behavior + +Set the following annotation on the performance profile: + +[source,yaml] +---- +metadata: + annotations: + performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable" +---- + +Valid values are as follows: + +* `disable`:: Exclude `ovsDpdk` CPUs from the kernel scheduler load-balancing pool. The Operator configures the OVS-DPDK cgroup partition as `isolated`. +* `enable`, or omit the annotation:: Keep the default behavior. The Operator configures the OVS-DPDK cgroup partition as `member`. + +== Why the control is an annotation + +Disabling load balancing on `ovsDpdk` CPUs requires matching support on the OVS side for full effect. The Node Tuning Operator exposes the control as an annotation so that you can enable the extra isolation when your OVS-DPDK deployment supports it, without coupling that optional step to the core `spec.cpu.ovsDpdk` field. + +[NOTE] +==== +This annotation controls kernel CPU load balancing for the `ovsDpdk` CPU set. It is not the same as OVN-Kubernetes dynamic CPU affinity management for OVS services. +==== diff --git a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc new file mode 100644 index 000000000000..748a5840a387 --- /dev/null +++ b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc @@ -0,0 +1,97 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: REFERENCE +[id="cnf-verifying-ovs-dpdk-cpu-isolation_{context}"] += Verifying OVS-DPDK CPU isolation + +[role="_abstract"] +After the performance profile is applied and the node has rebooted, verify that the Node Tuning Operator configured kernel parameters, kubelet reserved CPUs, IRQ affinity, and the OVS-DPDK cgroup hierarchy as expected. + +.Prerequisites + +* You configured a `PerformanceProfile` CR with `spec.cpu.ovsDpdk`. +* The target node has rebooted and the machine config pool update is complete. +* You have access to the cluster as a user with the `cluster-admin` role. + +.Procedure + +. Start a debug session on a node that matches the performance profile: ++ +[source,terminal] +---- +$ oc debug node/ +---- + +. Set `/host` as the root directory for the debug shell: ++ +[source,terminal] +---- +# chroot /host +---- + +. Verify that kernel boot parameters include the OVS-DPDK CPUs: ++ +[source,terminal] +---- +# cat /proc/cmdline +---- ++ +Confirm the following: + +* `isolcpus` includes the union of `isolated` and `ovsDpdk` CPUs. +* `nohz_full` and `rcu_nocbs` include the `ovsDpdk` CPUs. +* `systemd.cpu_affinity` does not include the `ovsDpdk` CPUs. + +. Verify that kubelet reserved system CPUs are the union of `reserved` and `ovsDpdk`: ++ +[source,terminal] +---- +# cat /etc/kubernetes/kubelet.conf | grep reservedSystemCPUs +---- ++ +The reported set must equal `reserved` ∪ `ovsDpdk`. + +. Verify that IRQ balancing excludes the OVS-DPDK CPUs: ++ +[source,terminal] +---- +# cat /etc/sysconfig/irqbalance +---- ++ +Confirm that `IRQBALANCE_BANNED_CPUS` includes a mask that covers the `ovsDpdk` CPUs. + +. Verify that the default SMP affinity mask excludes the OVS-DPDK CPUs: ++ +[source,terminal] +---- +# cat /proc/irq/default_smp_affinity +---- + +. Verify that the OVS-DPDK configuration script is present: ++ +[source,terminal] +---- +# stat /usr/local/bin/ovs-dpdk-cpus-configure.sh +---- + +. Verify the OVS-DPDK cgroup hierarchy and CPU assignment: ++ +[source,terminal] +---- +# cat /sys/fs/cgroup/ovs.slice/ovs-vswitchd.service/ovsdpdk.slice/cgroup.type +# cat /sys/fs/cgroup/ovs.slice/ovs-vswitchd.service/ovsdpdk.slice/cpuset.cpus.exclusive +# cat /sys/fs/cgroup/ovs.slice/ovs-vswitchd.service/ovsdpdk.slice/cpuset.cpus.partition +---- ++ +Confirm the following: + +* `cgroup.type` is `threaded`. +* `cpuset.cpus.exclusive` matches the `spec.cpu.ovsDpdk` value. +* `cpuset.cpus.partition` is `isolated` when `performance.openshift.io/cpu-load-balancing-ovs-dpdk` is set to `disable`, or `member` when the annotation is omitted or set to `enable`. + +[NOTE] +==== +These checks confirm that the platform configuration for OVS-DPDK CPU isolation is in place. Measuring OVS-DPDK dataplane throughput and packet loss requires your OVS-DPDK application test environment. +==== diff --git a/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc new file mode 100644 index 000000000000..eaf9bedbbd06 --- /dev/null +++ b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc @@ -0,0 +1,32 @@ +:_mod-docs-content-type: ASSEMBLY +[id="cnf-reserving-cpus-for-ovs-dpdk"] += Reserving CPUs for OVS-DPDK +include::_attributes/common-attributes.adoc[] +:context: cnf-reserving-cpus-for-ovs-dpdk + +toc::[] + +[role="_abstract"] +Reserve CPUs for Open vSwitch Data Plane Development Kit (OVS-DPDK) poll mode driver (PMD) threads by using a `PerformanceProfile` custom resource (CR). The Node Tuning Operator applies multi-layer CPU isolation so that those host-side threads can run with reduced interference from the operating system and Kubernetes scheduling. + +:FeatureName: Reserving CPUs for OVS-DPDK +include::snippets/technology-preview.adoc[] + +include::modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc[leveloffset=+1] + +include::modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc[leveloffset=+1] + +include::modules/cnf-configuring-ovs-dpdk-cpus.adoc[leveloffset=+1] + +include::modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc[leveloffset=+1] + +include::modules/cnf-performance-profile-ovs-dpdk-reference.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* xref:../scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc#cnf-create-performance-profiles_cnf-tuning-low-latency-nodes-with-perf-profile[Creating a performance profile] + +* xref:../scalability_and_performance/enabling-workload-partitioning.adoc#enabling-workload-partitioning[Workload partitioning] + +* xref:../scalability_and_performance/using-node-tuning-operator.adoc#using-node-tuning-operator[Using the Node Tuning Operator] From 190c484c60a71bb85021b6171c5b0f48bd9b4f7e Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Thu, 23 Jul 2026 16:16:34 +0100 Subject: [PATCH 2/6] TELCODOCS-2914: Apply JTBD Option B structure and drop callouts Reorder the OVS-DPDK assembly to do the job first, retitle modules without gerunds, expand the load-balancing rationale, and replace unsupported AsciiDoc callouts with a field list. Co-authored-by: Cursor --- _topic_maps/_topic_map.yml | 2 +- modules/cnf-configuring-ovs-dpdk-cpus.adoc | 25 +++++++++++-------- ...erformance-profile-ovs-dpdk-reference.adoc | 4 +-- ...-understanding-ovs-dpdk-cpu-isolation.adoc | 4 +-- ...rstanding-ovs-dpdk-cpu-load-balancing.adoc | 20 +++++++++++---- .../cnf-verifying-ovs-dpdk-cpu-isolation.adoc | 2 +- .../cnf-reserving-cpus-for-ovs-dpdk.adoc | 8 +++--- 7 files changed, 39 insertions(+), 26 deletions(-) diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index bd4f9fd15e6d..bb0e495f6e33 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -3517,7 +3517,7 @@ Topics: File: cnf-understanding-low-latency - Name: Tuning nodes for low latency with the performance profile File: cnf-tuning-low-latency-nodes-with-perf-profile -- Name: Reserving CPUs for OVS-DPDK +- Name: Reserve CPUs for OVS-DPDK File: cnf-reserving-cpus-for-ovs-dpdk Distros: openshift-origin,openshift-enterprise - Name: Tuning hosted control planes for low latency with the performance profile diff --git a/modules/cnf-configuring-ovs-dpdk-cpus.adoc b/modules/cnf-configuring-ovs-dpdk-cpus.adoc index e68a5a29174e..a8f451c5d616 100644 --- a/modules/cnf-configuring-ovs-dpdk-cpus.adoc +++ b/modules/cnf-configuring-ovs-dpdk-cpus.adoc @@ -4,10 +4,10 @@ :_mod-docs-content-type: PROCEDURE [id="cnf-configuring-ovs-dpdk-cpus_{context}"] -= Configuring OVS-DPDK CPUs in a performance profile += Reserve OVS-DPDK CPUs with a performance profile [role="_abstract"] -Create or update a `PerformanceProfile` custom resource (CR) that reserves CPUs for OVS-DPDK by setting `spec.cpu.ovsDpdk`. Optionally disable kernel load balancing on those CPUs with an annotation. +Create or update a `PerformanceProfile` custom resource (CR) that reserves CPUs for OVS-DPDK by setting `spec.cpu.ovsDpdk`. You can optionally add the load-balancing annotation in the same CR; see _Disable load balancing on OVS-DPDK CPUs_ for when that extra hardening is useful. .Prerequisites @@ -38,19 +38,22 @@ kind: PerformanceProfile metadata: name: ovs-dpdk-isolation annotations: - performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable" <1> + performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable" spec: cpu: - reserved: "0-1" <2> - ovsDpdk: "2-3" <3> - isolated: "4-7" <4> + reserved: "0-1" + ovsDpdk: "2-3" + isolated: "4-7" nodeSelector: node-role.kubernetes.io/worker: "" ---- -<1> Optional. Set to `disable` to exclude `ovsDpdk` CPUs from kernel load balancing and to configure the OVS-DPDK cgroup partition as `isolated`. Omit the annotation to keep the default `member` partition. -<2> CPUs for operating system and platform components. -<3> CPUs reserved for OVS-DPDK PMD threads. Do not reuse this set for unrelated DPDK applications. -<4> CPUs for Guaranteed application pods. ++ +where: + +* `performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"` is optional. Set it only when you need stronger scheduler isolation for PMD threads. Omit the annotation for the base reservation. For why and when to use this annotation, see _Disable load balancing on OVS-DPDK CPUs_. +* `spec.cpu.reserved` lists CPUs for operating system and platform components. +* `spec.cpu.ovsDpdk` lists CPUs reserved for OVS-DPDK PMD threads. Do not reuse this set for unrelated DPDK applications. +* `spec.cpu.isolated` lists CPUs for Guaranteed application pods. + [IMPORTANT] ==== @@ -85,4 +88,4 @@ Confirm that the `Available` condition is `True`. If the profile is `Degraded` w .Verification * Confirm that the `PerformanceProfile` status shows `Available=True`. -* Complete the checks in _Verifying OVS-DPDK CPU isolation_ to confirm node-level configuration. +* Complete the checks in _Verify OVS-DPDK CPU isolation_ to confirm node-level configuration. diff --git a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc index 1654d928e127..7f46166ef7cc 100644 --- a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc +++ b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc @@ -4,10 +4,10 @@ :_mod-docs-content-type: REFERENCE [id="cnf-performance-profile-ovs-dpdk-reference_{context}"] -= Performance profile fields for OVS-DPDK CPUs += OVS-DPDK performance profile fields [role="_abstract"] -Use the following reference when you configure OVS-DPDK CPU reservation in a `PerformanceProfile` custom resource (CR). +Use the following reference when you reserve OVS-DPDK CPUs in a `PerformanceProfile` custom resource (CR). == API fields and annotations diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc index 212599d4fded..9cd6b6b0640f 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc @@ -4,10 +4,10 @@ :_mod-docs-content-type: CONCEPT [id="cnf-understanding-ovs-dpdk-cpu-isolation_{context}"] -= Understanding OVS-DPDK CPU isolation += OVS-DPDK CPU isolation [role="_abstract"] -OVS-DPDK poll mode driver (PMD) threads run as host processes and continuously poll network interfaces. Any interruption from kernel interrupts, system daemons, or Kubernetes pods can introduce jitter and reduce throughput. Reserving CPUs with `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) gives those threads multi-layer isolation that is comparable to the isolation Guaranteed pods receive on isolated CPUs. +OVS-DPDK poll mode driver (PMD) threads run as host processes and need exclusive CPUs. Use `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) when reserved and isolated CPU sets alone cannot protect those threads from jitter. == Why reserved and isolated CPU sets are not enough diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc index f5ce8ec1e3ae..0efcbc95d82e 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc @@ -4,14 +4,24 @@ :_mod-docs-content-type: CONCEPT [id="cnf-understanding-ovs-dpdk-cpu-load-balancing_{context}"] -= Understanding the OVS-DPDK CPU load-balancing annotation += Disable load balancing on OVS-DPDK CPUs [role="_abstract"] -You can optionally exclude `ovsDpdk` CPUs from the kernel scheduler load-balancing pool by setting an annotation on the `PerformanceProfile` custom resource (CR). This control is separate from reserving the CPUs with `spec.cpu.ovsDpdk`. +After you reserve CPUs with `spec.cpu.ovsDpdk`, you can optionally disable kernel CPU load balancing on the same set. Use this when PMD threads still risk scheduler interference and your OVS-DPDK stack supports the matching OVS-side configuration. + +== Why disable load balancing on OVS-DPDK CPUs + +OVS-DPDK PMD threads typically pin to specific CPUs and busy-poll the NIC. Latency and packet loss increase when anything else runs on those CPUs, even briefly. + +The `spec.cpu.ovsDpdk` field applies multi-layer reservation: kubelet does not schedule pods on the CPUs, IRQ balancing excludes them, and the Node Tuning Operator places them under the OVS cgroup hierarchy. By default, those CPUs can still remain in the kernel scheduler load-balancing pool (`cpuset.cpus.partition=member`). + +Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). The advantage is stronger exclusive ownership for PMD threads: the kernel is less likely to migrate or run unrelated tasks on the same CPUs, which helps keep poll-loop timing stable. + +Use the annotation when you need that extra scheduler isolation and your OVS-DPDK stack supports the matching OVS-side configuration. If you only need the base reservation, set `spec.cpu.ovsDpdk` and omit the annotation. == Annotation behavior -Set the following annotation on the performance profile: +Set the following annotation on the `PerformanceProfile` custom resource (CR): [source,yaml] ---- @@ -27,9 +37,9 @@ Valid values are as follows: == Why the control is an annotation -Disabling load balancing on `ovsDpdk` CPUs requires matching support on the OVS side for full effect. The Node Tuning Operator exposes the control as an annotation so that you can enable the extra isolation when your OVS-DPDK deployment supports it, without coupling that optional step to the core `spec.cpu.ovsDpdk` field. +Full effect of disabling load balancing depends on matching support in the OVS process. The Node Tuning Operator exposes the control as an optional annotation so that base CPU reservation with `spec.cpu.ovsDpdk` stays independent of this extra hardening step. [NOTE] ==== -This annotation controls kernel CPU load balancing for the `ovsDpdk` CPU set. It is not the same as OVN-Kubernetes dynamic CPU affinity management for OVS services. +This annotation controls kernel CPU load balancing for the `ovsDpdk` CPU set. It is not the same as the pod annotation `cpu-load-balancing.crio.io`, and it is not the same as OVN-Kubernetes dynamic CPU affinity management for OVS services. ==== diff --git a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc index 748a5840a387..44213a2a18cc 100644 --- a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc @@ -4,7 +4,7 @@ :_mod-docs-content-type: REFERENCE [id="cnf-verifying-ovs-dpdk-cpu-isolation_{context}"] -= Verifying OVS-DPDK CPU isolation += Verify OVS-DPDK CPU isolation [role="_abstract"] After the performance profile is applied and the node has rebooted, verify that the Node Tuning Operator configured kernel parameters, kubelet reserved CPUs, IRQ affinity, and the OVS-DPDK cgroup hierarchy as expected. diff --git a/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc index eaf9bedbbd06..e889f648fee7 100644 --- a/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc +++ b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc @@ -1,6 +1,6 @@ :_mod-docs-content-type: ASSEMBLY [id="cnf-reserving-cpus-for-ovs-dpdk"] -= Reserving CPUs for OVS-DPDK += Reserve CPUs for OVS-DPDK include::_attributes/common-attributes.adoc[] :context: cnf-reserving-cpus-for-ovs-dpdk @@ -9,17 +9,17 @@ toc::[] [role="_abstract"] Reserve CPUs for Open vSwitch Data Plane Development Kit (OVS-DPDK) poll mode driver (PMD) threads by using a `PerformanceProfile` custom resource (CR). The Node Tuning Operator applies multi-layer CPU isolation so that those host-side threads can run with reduced interference from the operating system and Kubernetes scheduling. -:FeatureName: Reserving CPUs for OVS-DPDK +:FeatureName: OVS-DPDK CPU reservation include::snippets/technology-preview.adoc[] include::modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc[leveloffset=+1] -include::modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc[leveloffset=+1] - include::modules/cnf-configuring-ovs-dpdk-cpus.adoc[leveloffset=+1] include::modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc[leveloffset=+1] +include::modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc[leveloffset=+1] + include::modules/cnf-performance-profile-ovs-dpdk-reference.adoc[leveloffset=+1] [role="_additional-resources"] From 9c36772b1d42ee8263e2c1a33b9e1a053cf3e316 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Thu, 23 Jul 2026 16:51:38 +0100 Subject: [PATCH 3/6] TELCODOCS-2914: Add day-2 ovsDpdk guidance from QE test plan Document scheduling-domain isolation for the load-balancing annotation, expand verification expectations, and add update/remove procedures for changing or clearing spec.cpu.ovsDpdk. Co-authored-by: Cursor --- ...erformance-profile-ovs-dpdk-reference.adoc | 9 ++- ...rstanding-ovs-dpdk-cpu-load-balancing.adoc | 8 ++- ...nf-updating-or-removing-ovs-dpdk-cpus.adoc | 58 +++++++++++++++++++ .../cnf-verifying-ovs-dpdk-cpu-isolation.adoc | 2 + .../cnf-reserving-cpus-for-ovs-dpdk.adoc | 2 + 5 files changed, 74 insertions(+), 5 deletions(-) create mode 100644 modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc diff --git a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc index 7f46166ef7cc..bb18c1ddb8c5 100644 --- a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc +++ b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc @@ -21,7 +21,7 @@ Use the following reference when you reserve OVS-DPDK CPUs in a `PerformanceProf |`metadata.annotations["performance.openshift.io/cpu-load-balancing-ovs-dpdk"]` |string -|Optional. Set to `disable` to exclude `ovsDpdk` CPUs from kernel scheduler load balancing and to set the OVS-DPDK cgroup partition to `isolated`. Omit the annotation or set `enable` to keep the default `member` partition. +|Optional. Set to `disable` to exclude `ovsDpdk` CPUs from kernel scheduler load balancing and scheduling domains, and to set the OVS-DPDK cgroup partition to `isolated`. Omit the annotation or set `enable` to keep the default `member` partition. |=== == Validation and status @@ -30,6 +30,11 @@ Use the following reference when you reserve OVS-DPDK CPUs in a `PerformanceProf * Workload partitioning or kubelet `strict-cpu-reservation` is required. If the prerequisite is missing, the profile becomes `Degraded` with reason `OvsDpdkCPUsPrerequisiteNotMet`. * The Node Tuning Operator adds `ovsDpdk` CPUs to kubelet `reservedSystemCPUs` together with `reserved`, and to TuneD `isolated_cores` together with `isolated`. +== Day-2 changes + +* When you change `spec.cpu.ovsDpdk`, keep all CPU sets disjoint, wait for the MCO and node reboot, and re-verify isolation for the new set. CPUs that leave `ovsDpdk` must return to their new role without leftover IRQ bans or exclusive cgroup assignment. +* When you remove `spec.cpu.ovsDpdk` and the optional annotation, expect full cleanup of OVS-DPDK isolation artifacts, including `ovsdpdk.slice`. See _Update or remove OVS-DPDK CPU reservation_. + == Example configuration [source,yaml] @@ -54,4 +59,4 @@ spec: * Technology Preview support only. * Specific to OVS-DPDK. Not a generic dedicated-CPU pool for other DPDK applications. * The Performance Profile Creator (PPC) tool does not generate `spec.cpu.ovsDpdk` in this release. Edit the performance profile manually. -* Full effect of `performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"` depends on matching OVS-side support. +* Full effect of `performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"` depends on matching OVS-side support. \ No newline at end of file diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc index 0efcbc95d82e..c0eec93e17e4 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc @@ -15,7 +15,9 @@ OVS-DPDK PMD threads typically pin to specific CPUs and busy-poll the NIC. Laten The `spec.cpu.ovsDpdk` field applies multi-layer reservation: kubelet does not schedule pods on the CPUs, IRQ balancing excludes them, and the Node Tuning Operator places them under the OVS cgroup hierarchy. By default, those CPUs can still remain in the kernel scheduler load-balancing pool (`cpuset.cpus.partition=member`). -Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). The advantage is stronger exclusive ownership for PMD threads: the kernel is less likely to migrate or run unrelated tasks on the same CPUs, which helps keep poll-loop timing stable. +Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). Those CPUs are also removed from kernel scheduling domains. The exclusion is persistent for the `ovsDpdk` set: it does not depend on Guaranteed pod lifecycle the way the pod annotation `cpu-load-balancing.crio.io=disable` does. + +The advantage is stronger exclusive ownership for PMD threads: the kernel is less likely to migrate or run unrelated tasks on the same CPUs, which helps keep poll-loop timing stable. Use the annotation when you need that extra scheduler isolation and your OVS-DPDK stack supports the matching OVS-side configuration. If you only need the base reservation, set `spec.cpu.ovsDpdk` and omit the annotation. @@ -32,8 +34,8 @@ metadata: Valid values are as follows: -* `disable`:: Exclude `ovsDpdk` CPUs from the kernel scheduler load-balancing pool. The Operator configures the OVS-DPDK cgroup partition as `isolated`. -* `enable`, or omit the annotation:: Keep the default behavior. The Operator configures the OVS-DPDK cgroup partition as `member`. +* `disable`:: Exclude `ovsDpdk` CPUs from the kernel scheduler load-balancing pool and from kernel scheduling domains. The Operator configures the OVS-DPDK cgroup partition as `isolated`. +* `enable`, or omit the annotation:: Keep the default behavior. The Operator configures the OVS-DPDK cgroup partition as `member`. The CPUs remain reserved for OVS-DPDK, but they are not removed from scheduling domains by this annotation. == Why the control is an annotation diff --git a/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc new file mode 100644 index 000000000000..5981d09b240d --- /dev/null +++ b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc @@ -0,0 +1,58 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc + +:_mod-docs-content-type: PROCEDURE +[id="cnf-updating-or-removing-ovs-dpdk-cpus_{context}"] += Update or remove OVS-DPDK CPU reservation + +[role="_abstract"] +Change the `spec.cpu.ovsDpdk` CPU set on a day-2 retune, or remove the field when you no longer need OVS-DPDK CPU isolation. Keep CPU sets disjoint, wait for the Machine Config Operator (MCO) to apply changes, and re-verify isolation or cleanup on the node. + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. +* You have installed the OpenShift CLI (`oc`). +* A `PerformanceProfile` custom resource (CR) with `spec.cpu.ovsDpdk` is already applied. +* Workload partitioning is enabled, or the kubelet CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. + +.Procedure + +. To change the OVS-DPDK CPU set, edit the performance profile and replace `spec.cpu.ovsDpdk` with the new CPU list. Adjust `spec.cpu.isolated` and `spec.cpu.reserved` so that the sets do not overlap. ++ +.Example expand of the OVS-DPDK set +[source,yaml] +---- +spec: + cpu: + reserved: "0-1" + ovsDpdk: "2-3" + isolated: "4-7" +---- ++ +After the MCO applies the change and the node reboots, confirm that isolation reflects the new set: + +* Kernel parameters, `reservedSystemCPUs`, `IRQBALANCE_BANNED_CPUS`, and `ovsdpdk.slice` `cpuset.cpus.exclusive` include the new `ovsDpdk` CPUs. +* CPUs that left the `ovsDpdk` set are no longer banned or exclusively assigned for OVS-DPDK, and they match their new role in `isolated` or `reserved`. + +. To remove OVS-DPDK CPU reservation, edit the performance profile, delete `spec.cpu.ovsDpdk`, remove the `performance.openshift.io/cpu-load-balancing-ovs-dpdk` annotation if present, and move the former OVS-DPDK CPUs into `isolated` or another appropriate set so that your CPU layout remains valid. ++ +After the MCO applies the change and the node reboots, confirm cleanup: + +* `reservedSystemCPUs` no longer includes the former `ovsDpdk` CPUs (unless those CPUs were moved into `reserved`). +* `IRQBALANCE_BANNED_CPUS` and `default_smp_affinity` no longer isolate those CPUs for OVS-DPDK. +* The `ovsdpdk.slice` directory under `ovs.slice/ovs-vswitchd.service/` is gone. +* Former `ovsDpdk` CPUs behave according to their new role in the performance profile. + +. Apply the updated performance profile and wait for the machine config pool to finish updating: ++ +[source,terminal] +---- +$ oc apply -f performance-profile-ovs-dpdk.yaml +$ oc get mcp +---- + +.Verification + +* For an updated `ovsDpdk` set, complete the checks in _Verify OVS-DPDK CPU isolation_ against the new CPU list. +* For a removed `ovsDpdk` field, confirm that no OVS-DPDK isolation artifacts remain for the former CPU set. diff --git a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc index 44213a2a18cc..80f9b87cfa33 100644 --- a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc @@ -90,6 +90,8 @@ Confirm the following: * `cgroup.type` is `threaded`. * `cpuset.cpus.exclusive` matches the `spec.cpu.ovsDpdk` value. * `cpuset.cpus.partition` is `isolated` when `performance.openshift.io/cpu-load-balancing-ovs-dpdk` is set to `disable`, or `member` when the annotation is omitted or set to `enable`. ++ +When the annotation is set to `disable`, the `ovsDpdk` CPUs are also excluded from kernel scheduling domains. That exclusion must remain in place for the life of the reservation and must not change when unrelated Guaranteed pods start or stop on the node. [NOTE] ==== diff --git a/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc index e889f648fee7..a1f7362a55bd 100644 --- a/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc +++ b/scalability_and_performance/cnf-reserving-cpus-for-ovs-dpdk.adoc @@ -20,6 +20,8 @@ include::modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc[leveloffset=+1] include::modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc[leveloffset=+1] +include::modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc[leveloffset=+1] + include::modules/cnf-performance-profile-ovs-dpdk-reference.adoc[leveloffset=+1] [role="_additional-resources"] From 1287b8a20b8a7be2a7bb0dad5c649948d033a2b0 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Fri, 24 Jul 2026 11:57:19 +0100 Subject: [PATCH 4/6] TELCODOCS-2914: Tighten OVS-DPDK isolation concept for shortdesc and clarity Refactor the concept abstract and CPU-set table for DITA shortdesc limits, clearer isolation wording, and consistent QoS class monospace. Co-authored-by: Cursor --- modules/cnf-configuring-ovs-dpdk-cpus.adoc | 2 +- ...-understanding-ovs-dpdk-cpu-isolation.adoc | 20 +++++++++---------- ...rstanding-ovs-dpdk-cpu-load-balancing.adoc | 2 +- .../cnf-verifying-ovs-dpdk-cpu-isolation.adoc | 2 +- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/modules/cnf-configuring-ovs-dpdk-cpus.adoc b/modules/cnf-configuring-ovs-dpdk-cpus.adoc index a8f451c5d616..3e2c52e9e448 100644 --- a/modules/cnf-configuring-ovs-dpdk-cpus.adoc +++ b/modules/cnf-configuring-ovs-dpdk-cpus.adoc @@ -53,7 +53,7 @@ where: * `performance.openshift.io/cpu-load-balancing-ovs-dpdk: "disable"` is optional. Set it only when you need stronger scheduler isolation for PMD threads. Omit the annotation for the base reservation. For why and when to use this annotation, see _Disable load balancing on OVS-DPDK CPUs_. * `spec.cpu.reserved` lists CPUs for operating system and platform components. * `spec.cpu.ovsDpdk` lists CPUs reserved for OVS-DPDK PMD threads. Do not reuse this set for unrelated DPDK applications. -* `spec.cpu.isolated` lists CPUs for Guaranteed application pods. +* `spec.cpu.isolated` lists CPUs for `Guaranteed` application pods. + [IMPORTANT] ==== diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc index 9cd6b6b0640f..5602b099a2be 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc @@ -7,16 +7,16 @@ = OVS-DPDK CPU isolation [role="_abstract"] -OVS-DPDK poll mode driver (PMD) threads run as host processes and need exclusive CPUs. Use `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) when reserved and isolated CPU sets alone cannot protect those threads from jitter. +OVS-DPDK poll mode driver (PMD) threads busy-poll the NIC on the host and need exclusive CPUs. Use `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) when `reserved` and `isolated` alone cannot protect those threads from jitter. == Why reserved and isolated CPU sets are not enough -A performance profile already defines the following CPU sets: +OVS-DPDK PMD threads run as host processes outside the pod scheduling path. Interruption from the kernel, system daemons, or other workloads can introduce jitter and reduce throughput. A performance profile already defines the following CPU sets, which do not fully cover that case: -* `reserved`:: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, Burstable and BestEffort pods can still run on reserved CPUs. -* `isolated`:: CPUs for Guaranteed application pods that the kubelet schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK. +* `reserved`:: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, `Burstable` and `BestEffort` pods can still run on reserved CPUs. +* `isolated`:: CPUs for `Guaranteed` application pods that the kubelet schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK. -OVS-DPDK runs outside the pod scheduling path. To give PMD threads exclusive CPUs, the profile needs a third set that is excluded from kubelet scheduling and from host interference sources. +Because OVS-DPDK is not a `Guaranteed` pod workload, the profile needs a third CPU set that is excluded from kubelet scheduling and from host interference sources. == The three-tier CPU model for OVS-DPDK @@ -28,15 +28,15 @@ When you set `spec.cpu.ovsDpdk`, the Node Tuning Operator configures the followi |`reserved` |System daemons and platform components -|Excluded from application Guaranteed pods. Workload partitioning or related policies further protect these CPUs. +|`Guaranteed` application pods are not placed on these CPUs. Workload partitioning or related policies can further protect them. |`ovsDpdk` |OVS-DPDK PMD threads on the host -|Excluded from kubelet scheduling for all QoS classes, banned from `irqbalance`, excluded from systemd CPU affinity, and included in kernel isolation parameters. Placed under the OVS cgroup hierarchy. +|Pods of any QoS class are not placed on these CPUs. The Node Tuning Operator also bans them from `irqbalance`, removes them from systemd CPU affinity, adds them to kernel isolation parameters, and places them under the OVS cgroup hierarchy. |`isolated` -|Guaranteed latency-sensitive pods -|Available to the kubelet for Guaranteed workloads that request exclusive CPUs. +|`Guaranteed` latency-sensitive pods +|The kubelet can place `Guaranteed` workloads that request exclusive CPUs on this set. |=== [IMPORTANT] @@ -59,7 +59,7 @@ For CPUs listed in `spec.cpu.ovsDpdk`, the Node Tuning Operator applies configur Workload partitioning (`CPUPartitioningAllNodes`) or the kubelet CPU Manager `strict-cpu-reservation` policy option is required. If neither is enabled, the Node Tuning Operator marks the performance profile as `Degraded` with the reason `OvsDpdkCPUsPrerequisiteNotMet` and does not complete the configuration. -Without one of those prerequisites, Burstable and BestEffort pods can still be scheduled on the OVS-DPDK CPUs and break isolation. +Without one of those prerequisites, `Burstable` and `BestEffort` pods can still be scheduled on the OVS-DPDK CPUs and break isolation. [NOTE] ==== diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc index c0eec93e17e4..9dcdcbad88f4 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc @@ -15,7 +15,7 @@ OVS-DPDK PMD threads typically pin to specific CPUs and busy-poll the NIC. Laten The `spec.cpu.ovsDpdk` field applies multi-layer reservation: kubelet does not schedule pods on the CPUs, IRQ balancing excludes them, and the Node Tuning Operator places them under the OVS cgroup hierarchy. By default, those CPUs can still remain in the kernel scheduler load-balancing pool (`cpuset.cpus.partition=member`). -Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). Those CPUs are also removed from kernel scheduling domains. The exclusion is persistent for the `ovsDpdk` set: it does not depend on Guaranteed pod lifecycle the way the pod annotation `cpu-load-balancing.crio.io=disable` does. +Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). Those CPUs are also removed from kernel scheduling domains. The exclusion is persistent for the `ovsDpdk` set: it does not depend on `Guaranteed` pod lifecycle the way the pod annotation `cpu-load-balancing.crio.io=disable` does. The advantage is stronger exclusive ownership for PMD threads: the kernel is less likely to migrate or run unrelated tasks on the same CPUs, which helps keep poll-loop timing stable. diff --git a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc index 80f9b87cfa33..bf23fd258dc4 100644 --- a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc @@ -91,7 +91,7 @@ Confirm the following: * `cpuset.cpus.exclusive` matches the `spec.cpu.ovsDpdk` value. * `cpuset.cpus.partition` is `isolated` when `performance.openshift.io/cpu-load-balancing-ovs-dpdk` is set to `disable`, or `member` when the annotation is omitted or set to `enable`. + -When the annotation is set to `disable`, the `ovsDpdk` CPUs are also excluded from kernel scheduling domains. That exclusion must remain in place for the life of the reservation and must not change when unrelated Guaranteed pods start or stop on the node. +When the annotation is set to `disable`, the `ovsDpdk` CPUs are also excluded from kernel scheduling domains. That exclusion must remain in place for the life of the reservation and must not change when unrelated `Guaranteed` pods start or stop on the node. [NOTE] ==== From c05301a59be5fbc84b958aac9019d5632a782b11 Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Fri, 24 Jul 2026 15:28:57 +0100 Subject: [PATCH 5/6] Update based on cursor updates --- modules/cnf-configuring-ovs-dpdk-cpus.adoc | 2 +- .../cnf-performance-profile-ovs-dpdk-reference.adoc | 4 ++-- modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc | 10 +++++----- .../cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc | 2 +- modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc | 2 +- modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc | 4 ++-- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/modules/cnf-configuring-ovs-dpdk-cpus.adoc b/modules/cnf-configuring-ovs-dpdk-cpus.adoc index 3e2c52e9e448..3aa4f55ed37c 100644 --- a/modules/cnf-configuring-ovs-dpdk-cpus.adoc +++ b/modules/cnf-configuring-ovs-dpdk-cpus.adoc @@ -14,7 +14,7 @@ Create or update a `PerformanceProfile` custom resource (CR) that reserves CPUs * You have access to the cluster as a user with the `cluster-admin` role. * You have installed the OpenShift CLI (`oc`). * You have identified the CPU topology and NUMA locality of the OVS-DPDK NICs on the target nodes. -* Workload partitioning is enabled, or the kubelet CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. +* Workload partitioning is enabled, or the `kubelet` CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. * You understand that this feature is Technology Preview. .Procedure diff --git a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc index bb18c1ddb8c5..6db8edc9fed7 100644 --- a/modules/cnf-performance-profile-ovs-dpdk-reference.adoc +++ b/modules/cnf-performance-profile-ovs-dpdk-reference.adoc @@ -27,8 +27,8 @@ Use the following reference when you reserve OVS-DPDK CPUs in a `PerformanceProf == Validation and status * The `ovsDpdk` CPU set must not overlap the `reserved`, `isolated`, or `offlined` CPU sets. -* Workload partitioning or kubelet `strict-cpu-reservation` is required. If the prerequisite is missing, the profile becomes `Degraded` with reason `OvsDpdkCPUsPrerequisiteNotMet`. -* The Node Tuning Operator adds `ovsDpdk` CPUs to kubelet `reservedSystemCPUs` together with `reserved`, and to TuneD `isolated_cores` together with `isolated`. +* Workload partitioning or `kubelet` `strict-cpu-reservation` is required. If the prerequisite is missing, the profile becomes `Degraded` with reason `OvsDpdkCPUsPrerequisiteNotMet`. +* The Node Tuning Operator adds `ovsDpdk` CPUs to `kubelet` `reservedSystemCPUs` together with `reserved`, and to TuneD `isolated_cores` together with `isolated`. == Day-2 changes diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc index 5602b099a2be..d8e5b3c45796 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc @@ -14,9 +14,9 @@ OVS-DPDK poll mode driver (PMD) threads busy-poll the NIC on the host and need e OVS-DPDK PMD threads run as host processes outside the pod scheduling path. Interruption from the kernel, system daemons, or other workloads can introduce jitter and reduce throughput. A performance profile already defines the following CPU sets, which do not fully cover that case: * `reserved`:: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, `Burstable` and `BestEffort` pods can still run on reserved CPUs. -* `isolated`:: CPUs for `Guaranteed` application pods that the kubelet schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK. +* `isolated`:: CPUs for `Guaranteed` application pods that the `kubelet` schedules. Isolated CPUs are not intended for host processes such as OVS-DPDK. -Because OVS-DPDK is not a `Guaranteed` pod workload, the profile needs a third CPU set that is excluded from kubelet scheduling and from host interference sources. +Because OVS-DPDK is not a `Guaranteed` pod workload, the profile needs a third CPU set that is excluded from `kubelet` scheduling and from host interference sources. == The three-tier CPU model for OVS-DPDK @@ -36,7 +36,7 @@ When you set `spec.cpu.ovsDpdk`, the Node Tuning Operator configures the followi |`isolated` |`Guaranteed` latency-sensitive pods -|The kubelet can place `Guaranteed` workloads that request exclusive CPUs on this set. +|The `kubelet` can place `Guaranteed` workloads that request exclusive CPUs on this set. |=== [IMPORTANT] @@ -48,7 +48,7 @@ The `ovsDpdk` CPU set is specific to OVS-DPDK. Do not use it as a generic pool f For CPUs listed in `spec.cpu.ovsDpdk`, the Node Tuning Operator applies configuration across several layers, including the following: -* Adds the CPUs to the kubelet `reservedSystemCPUs` set together with `spec.cpu.reserved`. +* Adds the CPUs to the `kubelet` `reservedSystemCPUs` set together with `spec.cpu.reserved`. * Adds the CPUs to TuneD `isolated_cores` together with `spec.cpu.isolated`. * Includes the CPUs in kernel boot parameters such as `isolcpus`, `nohz_full`, and `rcu_nocbs`. * Excludes the CPUs from `systemd.cpu_affinity`. @@ -57,7 +57,7 @@ For CPUs listed in `spec.cpu.ovsDpdk`, the Node Tuning Operator applies configur == Prerequisites for isolation -Workload partitioning (`CPUPartitioningAllNodes`) or the kubelet CPU Manager `strict-cpu-reservation` policy option is required. If neither is enabled, the Node Tuning Operator marks the performance profile as `Degraded` with the reason `OvsDpdkCPUsPrerequisiteNotMet` and does not complete the configuration. +Workload partitioning (`CPUPartitioningAllNodes`) or the `kubelet` CPU Manager `strict-cpu-reservation` policy option is required. If neither is enabled, the Node Tuning Operator marks the performance profile as `Degraded` with the reason `OvsDpdkCPUsPrerequisiteNotMet` and does not complete the configuration. Without one of those prerequisites, `Burstable` and `BestEffort` pods can still be scheduled on the OVS-DPDK CPUs and break isolation. diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc index 9dcdcbad88f4..aec4c7714d58 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-load-balancing.adoc @@ -13,7 +13,7 @@ After you reserve CPUs with `spec.cpu.ovsDpdk`, you can optionally disable kerne OVS-DPDK PMD threads typically pin to specific CPUs and busy-poll the NIC. Latency and packet loss increase when anything else runs on those CPUs, even briefly. -The `spec.cpu.ovsDpdk` field applies multi-layer reservation: kubelet does not schedule pods on the CPUs, IRQ balancing excludes them, and the Node Tuning Operator places them under the OVS cgroup hierarchy. By default, those CPUs can still remain in the kernel scheduler load-balancing pool (`cpuset.cpus.partition=member`). +The `spec.cpu.ovsDpdk` field applies multi-layer reservation: `kubelet` does not schedule pods on the CPUs, IRQ balancing excludes them, and the Node Tuning Operator places them under the OVS cgroup hierarchy. By default, those CPUs can still remain in the kernel scheduler load-balancing pool (`cpuset.cpus.partition=member`). Disabling load balancing removes the CPUs from that shared scheduler pool (`cpuset.cpus.partition=isolated`). Those CPUs are also removed from kernel scheduling domains. The exclusion is persistent for the `ovsDpdk` set: it does not depend on `Guaranteed` pod lifecycle the way the pod annotation `cpu-load-balancing.crio.io=disable` does. diff --git a/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc index 5981d09b240d..c0b050159533 100644 --- a/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc +++ b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc @@ -14,7 +14,7 @@ Change the `spec.cpu.ovsDpdk` CPU set on a day-2 retune, or remove the field whe * You have access to the cluster as a user with the `cluster-admin` role. * You have installed the OpenShift CLI (`oc`). * A `PerformanceProfile` custom resource (CR) with `spec.cpu.ovsDpdk` is already applied. -* Workload partitioning is enabled, or the kubelet CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. +* Workload partitioning is enabled, or the `kubelet` CPU Manager `strict-cpu-reservation` policy option is configured on the target nodes. .Procedure diff --git a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc index bf23fd258dc4..9a2e7f297556 100644 --- a/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-verifying-ovs-dpdk-cpu-isolation.adoc @@ -7,7 +7,7 @@ = Verify OVS-DPDK CPU isolation [role="_abstract"] -After the performance profile is applied and the node has rebooted, verify that the Node Tuning Operator configured kernel parameters, kubelet reserved CPUs, IRQ affinity, and the OVS-DPDK cgroup hierarchy as expected. +After the performance profile is applied and the node has rebooted, verify that the Node Tuning Operator configured kernel parameters, `kubelet` reserved CPUs, IRQ affinity, and the OVS-DPDK cgroup hierarchy as expected. .Prerequisites @@ -44,7 +44,7 @@ Confirm the following: * `nohz_full` and `rcu_nocbs` include the `ovsDpdk` CPUs. * `systemd.cpu_affinity` does not include the `ovsDpdk` CPUs. -. Verify that kubelet reserved system CPUs are the union of `reserved` and `ovsDpdk`: +. Verify that `kubelet` reserved system CPUs are the union of `reserved` and `ovsDpdk`: + [source,terminal] ---- From defadd7835428ecd0b03cab29beec4b3b1d07f4f Mon Sep 17 00:00:00 2001 From: Kevin Quinn Date: Fri, 24 Jul 2026 17:05:02 +0100 Subject: [PATCH 6/6] improving writing --- modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc | 4 ---- modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc | 10 ++++++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc index d8e5b3c45796..489b5d967bda 100644 --- a/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc +++ b/modules/cnf-understanding-ovs-dpdk-cpu-isolation.adoc @@ -9,8 +9,6 @@ [role="_abstract"] OVS-DPDK poll mode driver (PMD) threads busy-poll the NIC on the host and need exclusive CPUs. Use `spec.cpu.ovsDpdk` in a `PerformanceProfile` custom resource (CR) when `reserved` and `isolated` alone cannot protect those threads from jitter. -== Why reserved and isolated CPU sets are not enough - OVS-DPDK PMD threads run as host processes outside the pod scheduling path. Interruption from the kernel, system daemons, or other workloads can introduce jitter and reduce throughput. A performance profile already defines the following CPU sets, which do not fully cover that case: * `reserved`:: CPUs for operating system daemons and Kubernetes infrastructure. Without additional controls, `Burstable` and `BestEffort` pods can still run on reserved CPUs. @@ -18,8 +16,6 @@ OVS-DPDK PMD threads run as host processes outside the pod scheduling path. Inte Because OVS-DPDK is not a `Guaranteed` pod workload, the profile needs a third CPU set that is excluded from `kubelet` scheduling and from host interference sources. -== The three-tier CPU model for OVS-DPDK - When you set `spec.cpu.ovsDpdk`, the Node Tuning Operator configures the following CPU roles on matching nodes: [cols="1,2,2"] diff --git a/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc index c0b050159533..337d86f70e53 100644 --- a/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc +++ b/modules/cnf-updating-or-removing-ovs-dpdk-cpus.adoc @@ -44,12 +44,18 @@ After the MCO applies the change and the node reboots, confirm cleanup: * The `ovsdpdk.slice` directory under `ovs.slice/ovs-vswitchd.service/` is gone. * Former `ovsDpdk` CPUs behave according to their new role in the performance profile. -. Apply the updated performance profile and wait for the machine config pool to finish updating: +. Apply the updated performance profile by running the following command + [source,terminal] ---- $ oc apply -f performance-profile-ovs-dpdk.yaml -$ oc get mcp +---- + +. Run the following command to check on the progress of the machine config updating and node reboot. Wait until the `MCP` is `Updated` and the node is `Ready` before proceeding: ++ +[source,terminal] +---- +$ oc get mcp -w ---- .Verification