For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

Multi-node NVIDIA B300 GPU worker nodes with eight GPUs in DigitalOcean Kubernetes (DOKS) clusters in the MKC1 datacenter connect to a rail-optimized GPU fabric. DOKS configures each fabric network interface controller (NIC) on these nodes automatically. To give Pods access to the fabric, you deploy the DigitalOcean-provided ip-preserving-host-device CNI plugin and use it in place of the standard host-device plugin.

This page covers only the Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) network configuration of 8-GPU B300 worker nodes in MKC1. Set up the rest of your multi-node GPU cluster, including the Multus CNI plugin, RDMA resource requests, and NCCL settings, as described in How to Configure Networking for Multi-Node GPU Worker Nodes, with the changes on this page.

Fabric Network Planes

The rail-optimized fabric is a Layer 3 routed network organized into two independent network planes. Each worker node has 16 fabric NICs, eight on each plane:

• Plane 0: The even-numbered interfaces, fabric0, fabric2, fabric4, fabric6, fabric8, fabric10, fabric12, and fabric14.
• Plane 1: The odd-numbered interfaces, fabric1, fabric3, fabric5, fabric7, fabric9, fabric11, fabric13, and fabric15.

Each GPU connects to one fabric NIC on each plane, with PCI affinity:

Within a plane, any two worker nodes in MKC1 can communicate across all rails. The two planes are isolated from each other, so a NIC on Plane 0 cannot reach a NIC on Plane 1.

Fabric NIC Configuration

DigitalOcean assigns each fabric NIC an IPv6 Unique Local Address (ULA), one /64 per interface in the fd02::/24 range, along with its routing information. The configuration is available from the metadata service at http://169.254.169.254/v1/gpu-fabric-nics.

When DOKS creates a worker node, it reads this configuration from the metadata service and configures each fabric NIC with its assigned IPv6 address. It also places each NIC in its own virtual routing and forwarding (VRF) instance with a dedicated routing table, which gives each NIC an isolated routing context for connectivity and path selection. You don’t need to configure the NICs yourself.

Verify the NIC Configuration

DOKS worker nodes don’t accept SSH connections. To inspect a worker node’s network configuration, start a debugging session on the node and switch to the node’s root file system. Replace <node-name> with the name of a B300 worker node from kubectl get nodes:

kubectl debug node/<node-name> -it --profile=sysadmin --image=ubuntu -- chroot /host

In the session, list the network interfaces:

ip -br a

Every fabric interface on both planes must be UP and have two IPv6 addresses: the ULA from the metadata service, and an automatically assigned link-local address in fe80::/10. The link-local address is not routable; do not use it for RDMA. The second 16-bit group of the ULA identifies the plane: fd02:0:... for Plane 0 and fd02:1:... for Plane 1. A VRF device for each interface, vrf-fabric0 through vrf-fabric15, must also exist and be UP. Your addresses differ from the following example:

Output
eth0             UP             203.0.113.95/20 fe80::70b8:40ff:fe56:bbc/64
eth1             UP             10.132.0.7/20 fe80::f8ce:1aff:fe78:ce74/64
fabric0          UP             fd02:0:400:527::1/64 fe80::ac3a:e2ff:fecf:d75e/64
fabric1          UP             fd02:1:400:527::1/64 fe80::ac3a:e2ff:fecf:d75f/64
fabric2          UP             fd02:0:500:527::1/64 fe80::ac3a:e2ff:fecf:d77e/64
fabric3          UP             fd02:1:500:527::1/64 fe80::ac3a:e2ff:fecf:d77f/64
fabric4          UP             fd02:0:600:527::1/64 fe80::ac3a:e2ff:fecf:d78e/64
fabric5          UP             fd02:1:600:527::1/64 fe80::ac3a:e2ff:fecf:d78f/64
fabric6          UP             fd02:0:700:527::1/64 fe80::ac3a:e2ff:fecf:d76e/64
fabric7          UP             fd02:1:700:527::1/64 fe80::ac3a:e2ff:fecf:d76f/64
fabric8          UP             fd02:0:300:527::1/64 fe80::ac3a:e2ff:fecf:d7ce/64
fabric9          UP             fd02:1:300:527::1/64 fe80::ac3a:e2ff:fecf:d7cf/64
fabric10         UP             fd02:0:200:527::1/64 fe80::ac3a:e2ff:fecf:d7ae/64
fabric11         UP             fd02:1:200:527::1/64 fe80::ac3a:e2ff:fecf:d7af/64
fabric12         UP             fd02:0:100:527::1/64 fe80::ac3a:e2ff:fecf:d79e/64
fabric13         UP             fd02:1:100:527::1/64 fe80::ac3a:e2ff:fecf:d79f/64
fabric14         UP             fd02:0:0:527::1/64 fe80::ac3a:e2ff:fecf:d7be/64
fabric15         UP             fd02:1:0:527::1/64 fe80::ac3a:e2ff:fecf:d7bf/64
...
vrf-fabric0      UP
vrf-fabric1      UP
vrf-fabric2      UP
vrf-fabric3      UP
vrf-fabric4      UP
vrf-fabric5      UP
vrf-fabric6      UP
vrf-fabric7      UP
vrf-fabric8      UP
vrf-fabric9      UP
vrf-fabric10     UP
vrf-fabric11     UP
vrf-fabric12     UP
vrf-fabric13     UP
vrf-fabric14     UP
vrf-fabric15     UP

A fabric NIC that a Pod currently holds does not appear on the node. Run these checks on a node with no Pods attached to the fabric, or inspect the attached NICs from inside the Pod.

The routing table of each VRF contains the directly connected /64 route for its fabric interface and a summary route to fd02::/24 through the leaf switch, which uses the ::2 address on the same /64. To check the routing table for fabric0, run:

ip -6 route show vrf vrf-fabric0

The output looks similar to the following:

Output
fd02:0:400:527::/64 dev fabric0 proto kernel metric 256 pref medium
fd02::/24 via fd02:0:400:527::2 dev fabric0 proto static metric 1024 onlink pref medium
fe80::/64 dev fabric0 proto kernel metric 256 pref medium
multicast ff00::/8 dev fabric0 proto kernel metric 256 pref medium

Verify Fabric Connectivity

First, verify that every fabric NIC can reach its gateway, the leaf switch at the ::2 address on the NIC’s /64. Run the ping from inside the NIC’s VRF. For example, for fabric0:

ip vrf exec vrf-fabric0 ping -c3 fd02:0:400:527::2

To check all 16 fabric NICs at once, run the following loop. All 16 pings must succeed:

for i in $(seq 0 15); do
  gw=$(ip -6 route show vrf vrf-fabric$i | awk '/via/ {print $3; exit}')
  ip vrf exec vrf-fabric$i ping -c3 "$gw"
done

Next, verify connectivity across rails and worker nodes within each plane. From each fabric NIC on this node, ping the ULA (the ::1 address) of each of the eight fabric NICs on the same plane of a peer worker node, for 64 pings per plane. All in-plane pings must succeed. For example, replace <peer-plane-0-ula> and <peer-plane-1-ula> with addresses from the peer node’s ip -br a output:

ip vrf exec vrf-fabric0 ping -c3 <peer-plane-0-ula>
ip vrf exec vrf-fabric15 ping -c3 <peer-plane-1-ula>

Pings between the two planes, such as from fabric0 to a peer’s fabric1, fail because the planes are isolated.

Deploy the ip-preserving-host-device CNI Plugin

The ip-preserving-host-device CNI plugin extends the standard host-device CNI plugin. Like host-device, it moves a fabric NIC from the node into a Pod’s network namespace. Unlike host-device, it preserves the NIC’s network configuration when it does: the IPv6 ULA, the link-local address, the NIC’s IPv6 routes, and its VRF and routing table. RDMA applications in the Pod can then use the same fabric addresses and routes as the node.

A DaemonSet installs the plugin on every worker node. Create a file named ip-preserving-host-device-cni.yaml:

nano ip-preserving-host-device-cni.yaml

Add the following contents:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ip-preserving-host-device-cni
  namespace: kube-system
  labels:
    app: ip-preserving-host-device-cni
spec:
  selector:
    matchLabels:
      app: ip-preserving-host-device-cni
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: ip-preserving-host-device-cni
    spec:
      priorityClassName: system-node-critical
      terminationGracePeriodSeconds: 5
      tolerations:
        - operator: Exists
      hostNetwork: true
      containers:
        - name: install
          image: ghcr.io/digitalocean-packages/ip-preserving-host-device:v1.0.0
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          volumeMounts:
            - name: cni-bin
              mountPath: /host/opt/cni/bin
            - name: cni-state
              mountPath: /var/lib/cni/ip-preserving-host-device
      volumes:
        - name: cni-bin
          hostPath:
            path: /opt/cni/bin
            type: DirectoryOrCreate
        - name: cni-state
          hostPath:
            path: /var/lib/cni/ip-preserving-host-device
            type: DirectoryOrCreate

Deploy the DaemonSet:

kubectl apply -f ip-preserving-host-device-cni.yaml

The DaemonSet copies the ip-preserving-host-device binary to /opt/cni/bin on each node. To confirm that the DaemonSet is running on every node, run:

kubectl rollout status daemonset/ip-preserving-host-device-cni --namespace=kube-system

Create NetworkAttachmentDefinitions for the Fabric NICs

NetworkAttachmentDefinition resources tell Multus CNI how to attach additional network interfaces to Pods. For the rail-optimized fabric, define one resource for each of the 16 fabric NICs, fabric0 through fabric15, with the ip-preserving-host-device plugin as the type instead of host-device. Create a file named nads.yaml:

nano nads.yaml

Add the following contents:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric0
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric0"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric1
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric1"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric2
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric2"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric3
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric3"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric4
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric4"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric5
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric5"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric6
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric6"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric7
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric7"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric8
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric8"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric9
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric9"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric10
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric10"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric11
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric11"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric12
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric12"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric13
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric13"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric14
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric14"
    }'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: roce-net-fabric15
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ip-preserving-host-device",
      "device": "fabric15"
    }'

Install the resources in the namespace where your RDMA workloads run:

kubectl apply -f nads.yaml --namespace=<your-namespace>

Attach the Fabric NICs to Pods

Reference the NetworkAttachmentDefinition resources in the k8s.v1.cni.cncf.io/networks annotation of your Pod or Deployment manifest, and request the matching rdma/fabric* resources, as described in Configure Multus CNI Plugin and Manage RDMA-Related Resources. Include all 16 fabric NICs, fabric0 through fabric15. For example, the annotation for all 16 NICs is:

metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: >-
      roce-net-fabric0@fabric0,
      roce-net-fabric1@fabric1,
      roce-net-fabric2@fabric2,
      roce-net-fabric3@fabric3,
      roce-net-fabric4@fabric4,
      roce-net-fabric5@fabric5,
      roce-net-fabric6@fabric6,
      roce-net-fabric7@fabric7,
      roce-net-fabric8@fabric8,
      roce-net-fabric9@fabric9,
      roce-net-fabric10@fabric10,
      roce-net-fabric11@fabric11,
      roce-net-fabric12@fabric12,
      roce-net-fabric13@fabric13,
      roce-net-fabric14@fabric14,
      roce-net-fabric15@fabric15

When a Pod that references these resources starts, each fabric NIC it references moves from the node into the Pod, along with its addresses, routes, and VRF. The NICs no longer appear on the node until the Pod is deleted, when they return to the node and become available to other Pods.

To confirm that a NIC kept its configuration, run ip -br a inside the Pod with kubectl exec. Each attached fabric* interface shows the same ULA and link-local address that it had on the node.

Configure RDMA Applications

Because each fabric NIC is bound to its own VRF, the fabric addresses are not reachable through the Pod’s default routing table. RDMA applications must either use the Pod’s primary interface, eth0, for connection establishment, or explicitly specify the source interface or VRF.

For NCCL configuration on B300 worker nodes, see Configure NCCL or RCCL. When you choose a GID index, use the RoCE v2 entry for the NIC’s ULA, not its link-local address. To list the GID entries for the attached NICs, run ibv_devinfo -v | grep GID inside the Pod. For more information, see GID Index Selection.