How to Configure Networking for Multi-Node GPU Worker Nodes

Last verified 1 Sep 2026

DigitalOcean Kubernetes (DOKS) is a Kubernetes service with a fully managed control plane, high availability, and autoscaling. DOKS integrates with standard Kubernetes toolchains and DigitalOcean’s load balancers, volumes, CPU and GPU Droplets, API, and CLI.

Copy page as Markdown View page as Markdown

Multi-node GPU clusters can only be created in multiples of 8 GPUs and are available by contract only. For more information on supported GPUs, see GPU Worker Nodes.

In a multi-node configuration, 8-GPU configurations are connected via a dedicated high-speed networking fabric in the DOKS cluster. The networking fabric is exposed on worker nodes through eight (or sixteen for NVIDIA B300) network interface controllers (NICs) named fabric0, fabric1, …, fabric7 (,fabric8, …, fabric15), which exist alongside the regular eth0 and eth1 interfaces. The eth0 interface provides public internet connectivity, and eth1 provides private connectivity to other nodes in the same VPC network. The fabric NICs enable AI/ML workloads to exchange data with very low latency and high throughput. To achieve high networking performance, we recommend using the Remote Direct Memory Access (RDMA) networking protocol for communication between the GPU nodes through the fabric NICs, which completely bypasses the CPU and kernel of the operating system for data transfer.

Additional plugins are required to enable the high-speed fabric for multi-node GPU networking. This guide covers the additional required components, how to configure them, and how to apply NCCL or RCCL settings in your workloads.

Required Plugins

To use the high-speed fabric with container-based workloads, the following Kubernetes plugins must be available on clusters with AMD or NVIDIA GPUs:

  • Mellanox k8s-rdma-shared-dev-plugin: This plugin is automatically installed in your DOKS cluster when you add a node pool with a fabric-connected slug. It exposes RDMA-related resources as Kubernetes resources, named rdma/fabric0, rdma/fabric1, rdma/fabric2, …. You can manage these resources using resource requests and limits in your manifests.

  • Multus CNI plugin: You must install this plugin manually. It moves the NICs fabric0, fabric1, ... into the container namespace via the host-device plugin. To install the plugin, run the following command:

kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
Note

Regular public and private communication in the clusters via eth0 and eth1 are not affected by the installation of the Multus CNI and continue to use Cilium.

After installing the CNI plugin, create NetworkAttachmentDefinition resources for the fabric NICs as described in the Configure Multus CNI Plugin section below.

Expose the RDMA-related resources managed by the Mellanox k8s-rdma-shared-dev-plugin to your workloads. To do this for AMD GPU nodes, add the following resource requests and limits to your Pod or Deployment manifest:

resources:
  requests:
    amd.com/gpu: 8
    rdma/fabric0: 1
    rdma/fabric1: 1
    rdma/fabric2: 1
    rdma/fabric3: 1
    rdma/fabric4: 1
    rdma/fabric5: 1
    rdma/fabric6: 1
    rdma/fabric7: 1
  limits:
    amd.com/gpu: 8
    rdma/fabric0: 1
    rdma/fabric1: 1
    rdma/fabric2: 1
    rdma/fabric3: 1
    rdma/fabric4: 1
    rdma/fabric5: 1
    rdma/fabric6: 1
    rdma/fabric7: 1

For NVIDIA GPU nodes, replace amd.com/gpu: 8 with nvidia.com/gpu: 8.

Configure Multus CNI Plugin

In addition to the RDMA-related resources, you must make the fabric NICs fabric0, fabric1, ... available to your containers. To do this, configure a set of NetworkAttachmentDefinition resources that use the host-device CNI plugin to expose each NIC.

Create a config file that contains the following fabric NICs:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric0
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric0"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric1
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric1"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric2
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric2"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric3
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric3"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric4
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric4"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric5
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric5"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric6
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric6"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric7
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "host-device",
      "device": "fabric7"
    }'

Remember that for NVIDIA B300, eight additional fabric8 to fabric15 stanzas are required. Install the resources in your desired namespace using the following command:

kubectl apply -f <your-manifest>.yaml --namespace=<your-namespace>

Next, make the fabric NICs available in your containers by adding an annotation to your Pod or Deployment manifest (similarly here, add fabric8 to fabric15 if using NVIDIA B300):

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: >-
      roce-net-fabric0@fabric0,
      roce-net-fabric1@fabric1,
      roce-net-fabric2@fabric2,
      roce-net-fabric3@fabric3,
      roce-net-fabric4@fabric4,
      roce-net-fabric5@fabric5,
      roce-net-fabric6@fabric6,
      roce-net-fabric7@fabric7

Use kubectl apply to apply the updates.

The k8s.v1.cni.cncf.io/networks annotation tells Multus to attach each fabric* interface to the Pod’s network namespace using your NetworkAttachmentDefinition resources and the host-device plugin. The interfaces attach after the Pod is scheduled and running and the CNI finishes attaching the devices.

Unattached fabric interfaces appear in the worker node’s default network namespace. Use ip link or a similar command to list them.

When Multus attaches an interface, it moves that interface into the Pod’s network namespace, so while a Pod holds a fabric interface, it does not appear on the host. Use kubectl exec to run ip link or ip addr inside the Pod to inspect attached interfaces. Do not use host-only checks alone to conclude RDMA or the fabric is missing while a Pod already uses those NICs.

You can also reference NetworkAttachmentDefinition from another namespace by using the namespace resource name in the annotation (for example, custom-namespace/roce-net-fabric0@fabric0). Each fabric NIC can only be attached to a single container at a time.

Once the fabric NICs are available in the containers, high-speed networking using RDMA is enabled between the GPU nodes.

Configure NCCL or RCCL

For the best performance with multi-node training, configure NCCL on NVIDIA GPUs or RCCL on AMD GPUs after the fabric NICs are attached to your Pods.

The recommended parameter sets for each GPU family, including the NVIDIA H100 topology file and GID index guidance, are documented in How to Configure Multi-Node GPU Droplets. Use those values in your DigitalOcean Kubernetes workloads, with one DOKS-specific change: set NCCL_SOCKET_IFNAME to eth0 (the Pod’s primary CNI interface), not eth1 as on Droplets.

You can apply the settings as environment variables on your containers, pass them with mpirun -x, or write them to /etc/nccl.conf (NVIDIA) or /etc/rccl.conf (AMD). The following examples show complete configurations for common multi-node GPU worker types.

NVIDIA H100 workers require a NCCL topology file in addition to the configuration parameters. Download the topology file from the NVIDIA H100 section of the Droplet multi-node guide and save it as /etc/nccl/topo.xml in your container image or at container start. Then create /etc/nccl.conf with the following contents:

NCCL_TOPO_FILE=/etc/nccl/topo.xml
NCCL_SOCKET_IFNAME==eth0
NCCL_CROSS_NIC=0
NCCL_NET_DISABLE_INTRA=1
NCCL_IB_TC=104
NCCL_DEBUG=WARN

You can create these files in an init container or at the start of your worker container command. For parameter descriptions, see NVIDIA H100 in the Droplet guide.

For NVIDIA H200 and B300 workers, set the following environment variables on your training containers. For B300, also attach fabric8 through fabric15 as described in Configure Multus CNI Plugin.

env:
  - name: NCCL_SOCKET_IFNAME
    value: eth0
  - name: NCCL_CROSS_NIC
    value: "0"
  - name: NCCL_NET_DISABLE_INTRA
    value: "1"
  - name: NCCL_IB_TC
    value: "104"

You can instead write the same settings to /etc/nccl.conf. For parameter descriptions, see NVIDIA H200 and NVIDIA B300 in the Droplet guide.

AMD Instinct MI300X and MI325X workers use the gfx942 architecture. Set the following environment variables on your training containers:

env:
  - name: NCCL_SOCKET_IFNAME
    value: eth0
  - name: NCCL_CROSS_NIC
    value: "0"
  - name: NCCL_PXN_DISABLE
    value: "0"
  - name: NCCL_NET_DISABLE_INTRA
    value: "1"
  - name: NCCL_IB_GID_INDEX
    value: "1"
  - name: NCCL_IB_TC
    value: "104"

RCCL reads the same NCCL_* variable names. You can instead write these settings to /etc/rccl.conf. You may need to update the NCCL_IB_GID_INDEX value for your environment; see GID Index Selection. For parameter descriptions, see AMD MI3XX Family in the Droplet guide.

AMD Instinct MI350X and later workers use the gfx950 architecture. Set the following environment variables on your training containers:

env:
  - name: NCCL_SOCKET_IFNAME
    value: eth0
  - name: NCCL_CROSS_NIC
    value: "0"
  - name: NCCL_PXN_DISABLE
    value: "0"
  - name: NCCL_NET_DISABLE_INTRA
    value: "1"
  - name: NCCL_IB_GID_INDEX
    value: "1"
  - name: NCCL_IB_TC
    value: "104"

RCCL reads the same NCCL_* variable names. You can instead write these settings to /etc/rccl.conf. You may need to update the NCCL_IB_GID_INDEX value for your environment; see GID Index Selection. For parameter descriptions, see AMD MI3XX Family in the Droplet guide.

If collective communication fails or you need more detail while debugging, set NCCL_DEBUG to INFO or TRACE instead of WARN. For more targeted logging, also set NCCL_DEBUG_SUBSYS=INIT,GRAPH,NET to collect additional detail for initialization, topology graph construction, and networking.

Example: Run a NCCL all_reduce Test on NVIDIA H100

The following example is a complete Kubeflow MPIJob that runs the NCCL Tests all_reduce_perf benchmark across two NVIDIA H100 worker nodes (16 GPUs total). It also showcases how to install the H100 NCCL topology file (topo.xml) and wire it through /etc/nccl.conf with NCCL_TOPO_FILE at worker start.

Before you apply the manifest:

Create a file named mpijob-multus-nvidia-h100.yaml and add the following. The worker container writes /etc/nccl/topo.xml and /etc/nccl.conf before starting sshd. The example includes a topology file for illustration. Always use the latest topo.xml from the NVIDIA H100 section of the Droplet multi-node guide and replace the embedded contents if they differ.

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: mpi-multus-nvidia
spec:
  slotsPerWorker: 8
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
            - name: mpi-launcher
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - mpirun
                - --allow-run-as-root
                - -np
                - "16"
                - -bind-to
                - none
                - -map-by
                - slot
                - -x
                - PATH
                - -mca
                - plm_rsh_args
                - "-p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
                - -mca
                - btl
                - self,tcp
                - -mca
                - pml
                - ^ucx
                - /opt/nccl-tests/build/all_reduce_perf
                - -b
                - 1G
                - -e
                - 16G
                - -f
                - "2"
                - -g
                - "1"
                - -w
                - "5"
                - --iters
                - "200"
    Worker:
      replicas: 2
      template:
        metadata:
          annotations:
            k8s.v1.cni.cncf.io/networks: >-
              roce-net-fabric0@fabric0,
              roce-net-fabric1@fabric1,
              roce-net-fabric2@fabric2,
              roce-net-fabric3@fabric3,
              roce-net-fabric4@fabric4,
              roce-net-fabric5@fabric5,
              roce-net-fabric6@fabric6,
              roce-net-fabric7@fabric7
        spec:
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          initContainers:
            - name: setup-ssh
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Setting up SSH configuration..."
                  mkdir -p /ssh-setup/sshd

                  echo "Generating SSH host keys..."
                  ssh-keygen -t rsa -f /ssh-setup/sshd/ssh_host_rsa_key -N '' 2>/dev/null
                  ssh-keygen -t ecdsa -f /ssh-setup/sshd/ssh_host_ecdsa_key -N '' 2>/dev/null
                  ssh-keygen -t ed25519 -f /ssh-setup/sshd/ssh_host_ed25519_key -N '' 2>/dev/null

                  cat > /ssh-setup/sshd/sshd_config << 'SSHD_EOF'
                  Port 2222
                  HostKey /etc/ssh-runtime/ssh_host_rsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ecdsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ed25519_key
                  PermitRootLogin yes
                  PubkeyAuthentication yes
                  AuthorizedKeysFile /root/.ssh/authorized_keys /root/.ssh/id_rsa.pub
                  PasswordAuthentication no
                  ChallengeResponseAuthentication no
                  UsePAM no
                  PrintMotd no
                  PidFile /var/run/sshd.pid
                  StrictModes no
                  SSHD_EOF

                  echo "SSH setup complete"
                  echo "Files in /ssh-setup/sshd:"
                  ls -la /ssh-setup/sshd/
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
          containers:
            - name: mpi-worker
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  mkdir -p /etc/nccl
                  cat << 'EOF' > /etc/nccl/topo.xml
                  <system version="1">
                    <cpu numaid="0" affinity="00000000,00000000,0000ffff,ffffffff,ffffffff" arch="x86_64" vendor="GenuineIntel" familyid="6" modelid="143">
                      <pci busid="ffff:ff:01.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:01:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:01:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_0" dev="0" speed="400000" port="1" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:02.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:02:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:02:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_1" dev="1" speed="400000" port="2" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:03.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:03:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:03:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_2" dev="2" speed="400000" port="3" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:04.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:04:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:04:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_3" dev="3" speed="400000" port="4" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                    </cpu>
                    <cpu numaid="1" affinity="ffffffff,ffffffff,ffff0000,0000000,00000000" arch="x86_64" vendor="GenuineIntel" familyid="6" modelid="143">
                      <pci busid="ffff:ff:05.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:05:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:05:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_4" dev="4" speed="400000" port="5" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:06.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:06:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:06:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_5" dev="5" speed="400000" port="6" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:07.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:07:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:07:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_6" dev="6" speed="400000" port="7" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                      <pci busid="ffff:ff:08.0" class="0x060400" link_speed="32 GT/s" link_width="16" vendor="0x0000" device="0x0000" subsystem_vendor="0x0000" subsystem_device="0x0000">
                        <pci busid="0000:08:01.0" class="0x030200" link_speed="32.0 GT/s PCIe" link_width="16"/>
                        <pci busid="0000:08:02.0" class="0x020000" link_speed="32.0 GT/s PCIe" link_width="16">
                          <nic>
                            <net name="mlx5_7" dev="7" speed="400000" port="8" latency="0.000000" maxconn="131072" gdr="1"/>
                          </nic>
                        </pci>
                      </pci>
                    </cpu>
                  </system>
                  EOF

                  cat << 'EOF' > /etc/nccl.conf
                  NCCL_TOPO_FILE=/etc/nccl/topo.xml
                  NCCL_SOCKET_IFNAME==eth0
                  NCCL_CROSS_NIC=0
                  NCCL_NET_DISABLE_INTRA=1
                  NCCL_IB_TC=104
                  NCCL_DEBUG=WARN
                  EOF

                  echo "Starting worker container..."
                  mkdir -p /var/run/sshd /etc/ssh-runtime

                  echo "Copying SSH host keys and config from init container..."
                  cp /ssh-setup/sshd/* /etc/ssh-runtime/

                  echo "SSH runtime files:"
                  ls -la /etc/ssh-runtime/

                  echo "Verifying operator-provided SSH keys at /root/.ssh..."
                  ls -la /root/.ssh/ || echo "Warning: /root/.ssh not found"

                  echo "Starting sshd in foreground..."
                  exec /usr/sbin/sshd -D -e -f /etc/ssh-runtime/sshd_config
              securityContext:
                privileged: true
                capabilities:
                  add:
                    - IPC_LOCK
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
                - name: dshm
                  mountPath: /dev/shm
              resources:
                limits:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
                requests:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
              readinessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 5
                periodSeconds: 3
                timeoutSeconds: 2
              livenessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 15
                periodSeconds: 10
                timeoutSeconds: 2
          volumes:
            - name: ssh-setup
              emptyDir: {}
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 16Gi

Apply the manifest:

kubectl apply -f mpijob-multus-nvidia-h100.yaml

The launcher runs /opt/nccl-tests/build/all_reduce_perf, which measures all-reduce bandwidth across the fabric. NCCL reads the topology from /etc/nccl/topo.xml through the NCCL_TOPO_FILE setting in /etc/nccl.conf. For the latest NCCL topology file, download topo.xml from the NVIDIA H100 section of the Droplet multi-node guide. Watch the launcher logs for the test output:

kubectl logs -l training.kubeflow.org/job-role=launcher -f

Expected Test Summary

A successful run ends with a summary like the following. Out of bounds values : 0 OK means the check passed. The average bus bandwidth value varies by GPU type and cluster configuration:

# Out of bounds values : 0 OK
# Avg bus bandwidth    : <avg-bus-bandwidth>
#
# Collective test concluded: all_reduce_perf

When the test finishes, delete the job:

kubectl delete mpijob mpi-multus-nvidia

Example: Run a NCCL all_reduce Test on NVIDIA H200

The following example is a complete Kubeflow MPIJob that runs the NCCL Tests all_reduce_perf benchmark across two NVIDIA H200 worker nodes (16 GPUs total). It attaches the fabric NICs with Multus, requests the RDMA resources, and passes the H200 NCCL parameters with mpirun -x. Unlike the H100 example, H200 does not require a custom topo.xml file.

Use the same Multus, MPI Operator, and cluster prerequisites as the H100 example, with at least two available multi-node H200 GPU worker nodes with free GPUs.

Create a file named mpijob-multus-nvidia-h200.yaml and add the following:

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: mpi-multus-nvidia
spec:
  slotsPerWorker: 8
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
            - name: mpi-launcher
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - mpirun
                - --allow-run-as-root
                - -np
                - "16"
                - -bind-to
                - none
                - -map-by
                - slot
                - -x
                - NCCL_SOCKET_IFNAME=eth0
                - -x
                - NCCL_CROSS_NIC=0
                - -x
                - NCCL_NET_DISABLE_INTRA=1
                - -x
                - NCCL_IB_TC=104
                - -x
                - NCCL_DEBUG=WARN
                - -x
                - PATH
                - -mca
                - plm_rsh_args
                - "-p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
                - -mca
                - btl
                - self,tcp
                - -mca
                - pml
                - ^ucx
                - /opt/nccl-tests/build/all_reduce_perf
                - -b
                - 1G
                - -e
                - 16G
                - -f
                - "2"
                - -g
                - "1"
                - -w
                - "5"
                - --iters
                - "200"
    Worker:
      replicas: 2
      template:
        metadata:
          annotations:
            k8s.v1.cni.cncf.io/networks: >-
              roce-net-fabric0@fabric0,
              roce-net-fabric1@fabric1,
              roce-net-fabric2@fabric2,
              roce-net-fabric3@fabric3,
              roce-net-fabric4@fabric4,
              roce-net-fabric5@fabric5,
              roce-net-fabric6@fabric6,
              roce-net-fabric7@fabric7
        spec:
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          initContainers:
            - name: setup-ssh
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Setting up SSH configuration..."
                  mkdir -p /ssh-setup/sshd

                  echo "Generating SSH host keys..."
                  ssh-keygen -t rsa -f /ssh-setup/sshd/ssh_host_rsa_key -N '' 2>/dev/null
                  ssh-keygen -t ecdsa -f /ssh-setup/sshd/ssh_host_ecdsa_key -N '' 2>/dev/null
                  ssh-keygen -t ed25519 -f /ssh-setup/sshd/ssh_host_ed25519_key -N '' 2>/dev/null

                  cat > /ssh-setup/sshd/sshd_config << 'SSHD_EOF'
                  Port 2222
                  HostKey /etc/ssh-runtime/ssh_host_rsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ecdsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ed25519_key
                  PermitRootLogin yes
                  PubkeyAuthentication yes
                  AuthorizedKeysFile /root/.ssh/authorized_keys /root/.ssh/id_rsa.pub
                  PasswordAuthentication no
                  ChallengeResponseAuthentication no
                  UsePAM no
                  PrintMotd no
                  PidFile /var/run/sshd.pid
                  StrictModes no
                  SSHD_EOF

                  echo "SSH setup complete"
                  echo "Files in /ssh-setup/sshd:"
                  ls -la /ssh-setup/sshd/
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
          containers:
            - name: mpi-worker
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Starting worker container..."
                  mkdir -p /var/run/sshd /etc/ssh-runtime

                  echo "Copying SSH host keys and config from init container..."
                  cp /ssh-setup/sshd/* /etc/ssh-runtime/

                  echo "SSH runtime files:"
                  ls -la /etc/ssh-runtime/

                  echo "Verifying operator-provided SSH keys at /root/.ssh..."
                  ls -la /root/.ssh/ || echo "Warning: /root/.ssh not found"

                  echo "Starting sshd in foreground..."
                  exec /usr/sbin/sshd -D -e -f /etc/ssh-runtime/sshd_config
              securityContext:
                privileged: true
                capabilities:
                  add:
                    - IPC_LOCK
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
                - name: dshm
                  mountPath: /dev/shm
              resources:
                limits:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
                requests:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
              readinessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 5
                periodSeconds: 3
                timeoutSeconds: 2
              livenessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 15
                periodSeconds: 10
                timeoutSeconds: 2
          volumes:
            - name: ssh-setup
              emptyDir: {}
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 16Gi

Apply the manifest:

kubectl apply -f mpijob-multus-nvidia-h200.yaml

The launcher runs /opt/nccl-tests/build/all_reduce_perf, which measures all-reduce bandwidth across the fabric. Watch the launcher logs for the test output:

kubectl logs -l training.kubeflow.org/job-role=launcher -f

A successful run ends with the same all_reduce_perf summary shown in Expected Test Summary.

When the test finishes, delete the job:

kubectl delete mpijob mpi-multus-nvidia

Example: Run a NCCL all_reduce Test on NVIDIA B300

The following example is a complete Kubeflow MPIJob that runs the NCCL Tests all_reduce_perf benchmark across two NVIDIA B300 worker nodes (16 GPUs total). B300 workers expose 16 fabric NICs (fabric0 through fabric15), so the Multus annotation and RDMA resource requests include all sixteen interfaces. Create the matching NetworkAttachmentDefinition resources for fabric8 through fabric15 in addition to fabric0 through fabric7, as described in Configure Multus CNI Plugin.

Use the same Multus, MPI Operator, and cluster prerequisites as the H100 example, with at least two available multi-node B300 GPU worker nodes with free GPUs. Like H200, B300 does not require a custom topo.xml file.

Create a file named mpijob-multus-nvidia-b300.yaml and add the following:

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: mpi-multus-nvidia
spec:
  slotsPerWorker: 8
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
            - name: mpi-launcher
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - mpirun
                - --allow-run-as-root
                - -np
                - "16"
                - -bind-to
                - none
                - -map-by
                - slot
                - -x
                - NCCL_SOCKET_IFNAME=eth0
                - -x
                - NCCL_CROSS_NIC=0
                - -x
                - NCCL_NET_DISABLE_INTRA=1
                - -x
                - NCCL_IB_TC=104
                - -x
                - NCCL_DEBUG=WARN
                - -x
                - PATH
                - -mca
                - plm_rsh_args
                - "-p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
                - -mca
                - btl
                - self,tcp
                - -mca
                - pml
                - ^ucx
                - /opt/nccl-tests/build/all_reduce_perf
                - -b
                - 1G
                - -e
                - 16G
                - -f
                - "2"
                - -g
                - "1"
                - -w
                - "5"
                - --iters
                - "200"
    Worker:
      replicas: 2
      template:
        metadata:
          annotations:
            k8s.v1.cni.cncf.io/networks: >-
              roce-net-fabric0@fabric0,
              roce-net-fabric1@fabric1,
              roce-net-fabric2@fabric2,
              roce-net-fabric3@fabric3,
              roce-net-fabric4@fabric4,
              roce-net-fabric5@fabric5,
              roce-net-fabric6@fabric6,
              roce-net-fabric7@fabric7,
              roce-net-fabric8@fabric8,
              roce-net-fabric9@fabric9,
              roce-net-fabric10@fabric10,
              roce-net-fabric11@fabric11,
              roce-net-fabric12@fabric12,
              roce-net-fabric13@fabric13,
              roce-net-fabric14@fabric14,
              roce-net-fabric15@fabric15
        spec:
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          initContainers:
            - name: setup-ssh
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Setting up SSH configuration..."
                  mkdir -p /ssh-setup/sshd

                  echo "Generating SSH host keys..."
                  ssh-keygen -t rsa -f /ssh-setup/sshd/ssh_host_rsa_key -N '' 2>/dev/null
                  ssh-keygen -t ecdsa -f /ssh-setup/sshd/ssh_host_ecdsa_key -N '' 2>/dev/null
                  ssh-keygen -t ed25519 -f /ssh-setup/sshd/ssh_host_ed25519_key -N '' 2>/dev/null

                  cat > /ssh-setup/sshd/sshd_config << 'SSHD_EOF'
                  Port 2222
                  HostKey /etc/ssh-runtime/ssh_host_rsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ecdsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ed25519_key
                  PermitRootLogin yes
                  PubkeyAuthentication yes
                  AuthorizedKeysFile /root/.ssh/authorized_keys /root/.ssh/id_rsa.pub
                  PasswordAuthentication no
                  ChallengeResponseAuthentication no
                  UsePAM no
                  PrintMotd no
                  PidFile /var/run/sshd.pid
                  StrictModes no
                  SSHD_EOF

                  echo "SSH setup complete"
                  echo "Files in /ssh-setup/sshd:"
                  ls -la /ssh-setup/sshd/
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
          containers:
            - name: mpi-worker
              image: ghcr.io/coreweave/nccl-tests:13.2.0-devel-ubuntu24.04-nccl2.29.7-1-7112046
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Starting worker container..."
                  mkdir -p /var/run/sshd /etc/ssh-runtime

                  echo "Copying SSH host keys and config from init container..."
                  cp /ssh-setup/sshd/* /etc/ssh-runtime/

                  echo "SSH runtime files:"
                  ls -la /etc/ssh-runtime/

                  echo "Verifying operator-provided SSH keys at /root/.ssh..."
                  ls -la /root/.ssh/ || echo "Warning: /root/.ssh not found"

                  echo "Starting sshd in foreground..."
                  exec /usr/sbin/sshd -D -e -f /etc/ssh-runtime/sshd_config
              securityContext:
                privileged: true
                capabilities:
                  add:
                    - IPC_LOCK
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
                - name: dshm
                  mountPath: /dev/shm
              resources:
                limits:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
                  rdma/fabric8: 1
                  rdma/fabric9: 1
                  rdma/fabric10: 1
                  rdma/fabric11: 1
                  rdma/fabric12: 1
                  rdma/fabric13: 1
                  rdma/fabric14: 1
                  rdma/fabric15: 1
                requests:
                  nvidia.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
                  rdma/fabric8: 1
                  rdma/fabric9: 1
                  rdma/fabric10: 1
                  rdma/fabric11: 1
                  rdma/fabric12: 1
                  rdma/fabric13: 1
                  rdma/fabric14: 1
                  rdma/fabric15: 1
              readinessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 5
                periodSeconds: 3
                timeoutSeconds: 2
              livenessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 15
                periodSeconds: 10
                timeoutSeconds: 2
          volumes:
            - name: ssh-setup
              emptyDir: {}
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 16Gi

Apply the manifest:

kubectl apply -f mpijob-multus-nvidia-b300.yaml

The launcher runs /opt/nccl-tests/build/all_reduce_perf, which measures all-reduce bandwidth across the fabric. Watch the launcher logs for the test output:

kubectl logs -l training.kubeflow.org/job-role=launcher -f

A successful run ends with the same all_reduce_perf summary shown in Expected Test Summary.

When the test finishes, delete the job:

kubectl delete mpijob mpi-multus-nvidia

Example: Run a RCCL all_reduce Test on AMD MI300X and MI325X

The following example is a complete Kubeflow MPIJob that runs the RCCL Tests all_reduce_perf benchmark across two AMD Instinct MI300X or MI325X worker nodes (16 GPUs total). These GPUs use the gfx942 architecture. The launcher passes the RCCL parameters recommended for this family with mpirun -x.

Build a container image that includes RCCL Tests before you run the job. Follow AMD’s Using RCCL with Docker instructions, then push the image to a registry your cluster can pull from. Replace <your-rccl-tests-image> in the manifest with that image reference.

Use the same Multus, MPI Operator, and cluster prerequisites as the H100 example, with at least two available multi-node MI300X or MI325X GPU worker nodes with free GPUs.

Create a file named mpijob-multus-amd-gfx942.yaml and add the following. Set doks.digitalocean.com/gpu-model to mi300x or mi325x to match your worker nodes. The example sets NCCL_IB_GID_INDEX=1; you may need to update that value for your environment. See GID Index Selection.

apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: mpi-multus-gfx942
spec:
  slotsPerWorker: 8
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
            - name: mpi-launcher
              image: <your-rccl-tests-image>
              command:
                - mpirun
                - --allow-run-as-root
                - -np
                - "16"
                - -bind-to
                - none
                - -map-by
                - slot
                - -x
                - NCCL_SOCKET_IFNAME=eth0
                - -x
                - NCCL_CROSS_NIC=0
                - -x
                - NCCL_PXN_DISABLE=0
                - -x
                - NCCL_NET_DISABLE_INTRA=1
                - -x
                - NCCL_IB_GID_INDEX=1
                - -x
                - NCCL_IB_TC=104
                - -x
                - NCCL_DEBUG=WARN
                - -x
                - PATH
                - -mca
                - plm_rsh_args
                - "-p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
                - -mca
                - btl
                - self,tcp
                - -mca
                - pml
                - ^ucx
                - /workspace/rccl-tests/build/all_reduce_perf
                - -b
                - 1G
                - -e
                - 16G
                - -f
                - "2"
                - -g
                - "1"
                - -w
                - "5"
                - --iters
                - "200"
    Worker:
      replicas: 2
      template:
        metadata:
          annotations:
            k8s.v1.cni.cncf.io/networks: >-
              roce-net-fabric0@fabric0,
              roce-net-fabric1@fabric1,
              roce-net-fabric2@fabric2,
              roce-net-fabric3@fabric3,
              roce-net-fabric4@fabric4,
              roce-net-fabric5@fabric5,
              roce-net-fabric6@fabric6,
              roce-net-fabric7@fabric7
        spec:
          nodeSelector:
            doks.digitalocean.com/gpu-model: mi325x
          tolerations:
            - key: "amd.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          initContainers:
            - name: setup-ssh
              image: <your-rccl-tests-image>
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Setting up SSH configuration..."
                  mkdir -p /ssh-setup/sshd

                  echo "Generating SSH host keys..."
                  ssh-keygen -t rsa -f /ssh-setup/sshd/ssh_host_rsa_key -N '' 2>/dev/null
                  ssh-keygen -t ecdsa -f /ssh-setup/sshd/ssh_host_ecdsa_key -N '' 2>/dev/null
                  ssh-keygen -t ed25519 -f /ssh-setup/sshd/ssh_host_ed25519_key -N '' 2>/dev/null

                  cat > /ssh-setup/sshd/sshd_config << 'SSHD_EOF'
                  Port 2222
                  HostKey /etc/ssh-runtime/ssh_host_rsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ecdsa_key
                  HostKey /etc/ssh-runtime/ssh_host_ed25519_key
                  PermitRootLogin yes
                  PubkeyAuthentication yes
                  AuthorizedKeysFile /root/.ssh/authorized_keys /root/.ssh/id_rsa.pub
                  PasswordAuthentication no
                  ChallengeResponseAuthentication no
                  UsePAM no
                  PrintMotd no
                  PidFile /var/run/sshd.pid
                  StrictModes no
                  SSHD_EOF

                  echo "SSH setup complete"
                  echo "Files in /ssh-setup/sshd:"
                  ls -la /ssh-setup/sshd/
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
          containers:
            - name: mpi-worker
              image: <your-rccl-tests-image>
              command:
                - /bin/bash
                - -c
                - |
                  set -ex

                  echo "Starting worker container..."
                  mkdir -p /var/run/sshd /etc/ssh-runtime

                  echo "Copying SSH host keys and config from init container..."
                  cp /ssh-setup/sshd/* /etc/ssh-runtime/

                  echo "SSH runtime files:"
                  ls -la /etc/ssh-runtime/

                  echo "Verifying operator-provided SSH keys at /root/.ssh..."
                  ls -la /root/.ssh/ || echo "Warning: /root/.ssh not found"

                  echo "Starting sshd in foreground..."
                  exec /usr/sbin/sshd -D -e -f /etc/ssh-runtime/sshd_config
              securityContext:
                privileged: true
                capabilities:
                  add:
                    - IPC_LOCK
              volumeMounts:
                - name: ssh-setup
                  mountPath: /ssh-setup
                - name: dshm
                  mountPath: /dev/shm
              resources:
                limits:
                  amd.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
                requests:
                  amd.com/gpu: 8
                  rdma/fabric0: 1
                  rdma/fabric1: 1
                  rdma/fabric2: 1
                  rdma/fabric3: 1
                  rdma/fabric4: 1
                  rdma/fabric5: 1
                  rdma/fabric6: 1
                  rdma/fabric7: 1
              readinessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 5
                periodSeconds: 3
                timeoutSeconds: 2
              livenessProbe:
                tcpSocket:
                  port: 2222
                initialDelaySeconds: 15
                periodSeconds: 10
                timeoutSeconds: 2
          volumes:
            - name: ssh-setup
              emptyDir: {}
            - name: dshm
              emptyDir:
                medium: Memory
                sizeLimit: 16Gi

Apply the manifest:

kubectl apply -f mpijob-multus-amd-gfx942.yaml

The launcher runs /workspace/rccl-tests/build/all_reduce_perf, which measures all-reduce bandwidth across the fabric. Watch the launcher logs for the test output:

kubectl logs -l training.kubeflow.org/job-role=launcher -f

A successful run ends with the same all_reduce_perf summary shown in Expected Test Summary.

When the test finishes, delete the job:

kubectl delete mpijob mpi-multus-gfx942

Example: Run a RCCL all_reduce Test on AMD MI350X and Later

The MPIJob for AMD Instinct MI350X and later workers is identical to the MI300X and MI325X example. These GPUs use the gfx950 architecture, so when you build the RCCL Tests Docker image, target gfx950 instead of gfx942. The image must also include the libionic1 package for fabric networking to function correctly. See AMD’s Using RCCL with Docker for build instructions.

Set doks.digitalocean.com/gpu-model in the worker nodeSelector to match your MI350X (or later) worker nodes.

We can't find any results for your search.

Try using different keywords or simplifying your search terms.