--- title: How to Configure Multi-Node B300 GPU Worker Nodes on the Rail-Optimized Fabric description: Verify the RDMA network interfaces on multi-node NVIDIA B300 GPU worker nodes in MKC1, which connect to a rail-optimized GPU fabric, and attach them … product: Kubernetes url: https://docs.digitalocean.com/products/kubernetes/how-to/configure-rail-optimized-gpu-fabric/ last_updated: "2026-11-04" --- > **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document). # How to Configure Multi-Node B300 GPU Worker Nodes on the Rail-Optimized Fabric DigitalOcean Kubernetes (DOKS) is a Kubernetes service with a fully managed control plane, high availability, and autoscaling. DOKS integrates with standard Kubernetes toolchains and DigitalOcean’s load balancers, volumes, CPU and GPU Droplets, API, and CLI. Multi-node NVIDIA B300 GPU worker nodes with eight GPUs in DigitalOcean Kubernetes (DOKS) clusters in the MKC1 datacenter connect to a rail-optimized GPU fabric. DOKS configures each fabric network interface controller (NIC) on these nodes automatically. To give Pods access to the fabric, you deploy the DigitalOcean-provided `ip-preserving-host-device` CNI plugin and use it in place of the standard `host-device` plugin. This page covers only the Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) network configuration of 8-GPU B300 worker nodes in MKC1. Set up the rest of your multi-node GPU cluster, including the Multus CNI plugin, RDMA resource requests, and NCCL settings, as described in [How to Configure Networking for Multi-Node GPU Worker Nodes](https://docs.digitalocean.com/products/kubernetes/how-to/configure-multinode-gpus/index.html.md), with the changes on this page. ## Fabric Network Planes The rail-optimized fabric is a Layer 3 routed network organized into two independent network planes. Each worker node has 16 fabric NICs, eight on each plane: - **Plane 0**: The even-numbered interfaces, `fabric0`, `fabric2`, `fabric4`, `fabric6`, `fabric8`, `fabric10`, `fabric12`, and `fabric14`. - **Plane 1**: The odd-numbered interfaces, `fabric1`, `fabric3`, `fabric5`, `fabric7`, `fabric9`, `fabric11`, `fabric13`, and `fabric15`. Each GPU connects to one fabric NIC on each plane, with PCI affinity: | GPU | Plane 0 NIC | Plane 1 NIC | |---|---|---| | 0 | `fabric0` | `fabric1` | | 1 | `fabric2` | `fabric3` | | 2 | `fabric4` | `fabric5` | | 3 | `fabric6` | `fabric7` | | 4 | `fabric8` | `fabric9` | | 5 | `fabric10` | `fabric11` | | 6 | `fabric12` | `fabric13` | | 7 | `fabric14` | `fabric15` | Within a plane, any two worker nodes in MKC1 can communicate across all rails. The two planes are isolated from each other, so a NIC on Plane 0 cannot reach a NIC on Plane 1. ## Fabric NIC Configuration DigitalOcean assigns each fabric NIC an IPv6 Unique Local Address (ULA), one `/64` per interface in the `fd02::/24` range, along with its routing information. The configuration is available from the metadata service at `http://169.254.169.254/v1/gpu-fabric-nics`. When DOKS creates a worker node, it reads this configuration from the metadata service and configures each fabric NIC with its assigned IPv6 address. It also places each NIC in its own virtual routing and forwarding (VRF) instance with a dedicated routing table, which gives each NIC an isolated routing context for connectivity and path selection. You don’t need to configure the NICs yourself. ## Verify the NIC Configuration DOKS worker nodes don’t accept SSH connections. To inspect a worker node’s network configuration, start a debugging session on the node and switch to the node’s root file system. Replace `` with the name of a B300 worker node from `kubectl get nodes`: ```shell kubectl debug node/ -it --profile=sysadmin --image=ubuntu -- chroot /host ``` In the session, list the network interfaces: ```shell ip -br a ``` Every fabric interface on both planes must be `UP` and have two IPv6 addresses: the ULA from the metadata service, and an automatically assigned link-local address in `fe80::/10`. The link-local address is not routable; do not use it for RDMA. The second 16-bit group of the ULA identifies the plane: `fd02:0:...` for Plane 0 and `fd02:1:...` for Plane 1. A VRF device for each interface, `vrf-fabric0` through `vrf-fabric15`, must also exist and be `UP`. Your addresses differ from the following example: ```text Output eth0 UP 203.0.113.95/20 fe80::70b8:40ff:fe56:bbc/64 eth1 UP 10.132.0.7/20 fe80::f8ce:1aff:fe78:ce74/64 fabric0 UP fd02:0:400:527::1/64 fe80::ac3a:e2ff:fecf:d75e/64 fabric1 UP fd02:1:400:527::1/64 fe80::ac3a:e2ff:fecf:d75f/64 fabric2 UP fd02:0:500:527::1/64 fe80::ac3a:e2ff:fecf:d77e/64 fabric3 UP fd02:1:500:527::1/64 fe80::ac3a:e2ff:fecf:d77f/64 fabric4 UP fd02:0:600:527::1/64 fe80::ac3a:e2ff:fecf:d78e/64 fabric5 UP fd02:1:600:527::1/64 fe80::ac3a:e2ff:fecf:d78f/64 fabric6 UP fd02:0:700:527::1/64 fe80::ac3a:e2ff:fecf:d76e/64 fabric7 UP fd02:1:700:527::1/64 fe80::ac3a:e2ff:fecf:d76f/64 fabric8 UP fd02:0:300:527::1/64 fe80::ac3a:e2ff:fecf:d7ce/64 fabric9 UP fd02:1:300:527::1/64 fe80::ac3a:e2ff:fecf:d7cf/64 fabric10 UP fd02:0:200:527::1/64 fe80::ac3a:e2ff:fecf:d7ae/64 fabric11 UP fd02:1:200:527::1/64 fe80::ac3a:e2ff:fecf:d7af/64 fabric12 UP fd02:0:100:527::1/64 fe80::ac3a:e2ff:fecf:d79e/64 fabric13 UP fd02:1:100:527::1/64 fe80::ac3a:e2ff:fecf:d79f/64 fabric14 UP fd02:0:0:527::1/64 fe80::ac3a:e2ff:fecf:d7be/64 fabric15 UP fd02:1:0:527::1/64 fe80::ac3a:e2ff:fecf:d7bf/64 ... vrf-fabric0 UP vrf-fabric1 UP vrf-fabric2 UP vrf-fabric3 UP vrf-fabric4 UP vrf-fabric5 UP vrf-fabric6 UP vrf-fabric7 UP vrf-fabric8 UP vrf-fabric9 UP vrf-fabric10 UP vrf-fabric11 UP vrf-fabric12 UP vrf-fabric13 UP vrf-fabric14 UP vrf-fabric15 UP ``` A fabric NIC that a Pod currently holds does not appear on the node. Run these checks on a node with no Pods attached to the fabric, or inspect the attached NICs from inside the Pod. The routing table of each VRF contains the directly connected `/64` route for its fabric interface and a summary route to `fd02::/24` through the leaf switch, which uses the `::2` address on the same `/64`. To check the routing table for `fabric0`, run: ```shell ip -6 route show vrf vrf-fabric0 ``` The output looks similar to the following: ```text Output fd02:0:400:527::/64 dev fabric0 proto kernel metric 256 pref medium fd02::/24 via fd02:0:400:527::2 dev fabric0 proto static metric 1024 onlink pref medium fe80::/64 dev fabric0 proto kernel metric 256 pref medium multicast ff00::/8 dev fabric0 proto kernel metric 256 pref medium ``` ## Verify Fabric Connectivity First, verify that every fabric NIC can reach its gateway, the leaf switch at the `::2` address on the NIC’s `/64`. Run the ping from inside the NIC’s VRF. For example, for `fabric0`: ```shell ip vrf exec vrf-fabric0 ping -c3 fd02:0:400:527::2 ``` To check all 16 fabric NICs at once, run the following loop. All 16 pings must succeed: ```shell for i in $(seq 0 15); do gw=$(ip -6 route show vrf vrf-fabric$i | awk '/via/ {print $3; exit}') ip vrf exec vrf-fabric$i ping -c3 "$gw" done ``` Next, verify connectivity across rails and worker nodes within each plane. From each fabric NIC on this node, ping the ULA (the `::1` address) of each of the eight fabric NICs on the same plane of a peer worker node, for 64 pings per plane. All in-plane pings must succeed. For example, replace `` and `` with addresses from the peer node’s `ip -br a` output: ```shell ip vrf exec vrf-fabric0 ping -c3 ip vrf exec vrf-fabric15 ping -c3 ``` Pings between the two planes, such as from `fabric0` to a peer’s `fabric1`, fail because the planes are isolated. ## Deploy the `ip-preserving-host-device` CNI Plugin The `ip-preserving-host-device` CNI plugin extends the standard [`host-device` CNI plugin](https://www.cni.dev/plugins/current/main/host-device/). Like `host-device`, it moves a fabric NIC from the node into a Pod’s network namespace. Unlike `host-device`, it preserves the NIC’s network configuration when it does: the IPv6 ULA, the link-local address, the NIC’s IPv6 routes, and its VRF and routing table. RDMA applications in the Pod can then use the same fabric addresses and routes as the node. A DaemonSet installs the plugin on every worker node. Create a file named `ip-preserving-host-device-cni.yaml`: ```shell nano ip-preserving-host-device-cni.yaml ``` Add the following contents: ip-preserving-host-device-cni.yaml ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: ip-preserving-host-device-cni namespace: kube-system labels: app: ip-preserving-host-device-cni spec: selector: matchLabels: app: ip-preserving-host-device-cni updateStrategy: type: RollingUpdate template: metadata: labels: app: ip-preserving-host-device-cni spec: priorityClassName: system-node-critical terminationGracePeriodSeconds: 5 tolerations: - operator: Exists hostNetwork: true containers: - name: install image: ghcr.io/digitalocean-packages/ip-preserving-host-device:v1.0.0 imagePullPolicy: IfNotPresent securityContext: privileged: true volumeMounts: - name: cni-bin mountPath: /host/opt/cni/bin - name: cni-state mountPath: /var/lib/cni/ip-preserving-host-device volumes: - name: cni-bin hostPath: path: /opt/cni/bin type: DirectoryOrCreate - name: cni-state hostPath: path: /var/lib/cni/ip-preserving-host-device type: DirectoryOrCreate ``` Deploy the DaemonSet: ```shell kubectl apply -f ip-preserving-host-device-cni.yaml ``` The DaemonSet copies the `ip-preserving-host-device` binary to `/opt/cni/bin` on each node. To confirm that the DaemonSet is running on every node, run: ```shell kubectl rollout status daemonset/ip-preserving-host-device-cni --namespace=kube-system ``` ## Create NetworkAttachmentDefinitions for the Fabric NICs `NetworkAttachmentDefinition` resources tell Multus CNI how to attach additional network interfaces to Pods. For the rail-optimized fabric, define one resource for each of the 16 fabric NICs, `fabric0` through `fabric15`, with the `ip-preserving-host-device` plugin as the type instead of `host-device`. Create a file named `nads.yaml`: ```shell nano nads.yaml ``` Add the following contents: nads.yaml ```yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric0 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric0" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric1 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric1" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric2 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric2" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric3 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric3" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric4 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric4" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric5 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric5" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric6 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric6" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric7 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric7" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric8 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric8" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric9 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric9" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric10 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric10" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric11 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric11" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric12 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric12" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric13 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric13" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric14 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric14" }' --- apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: roce-net-fabric15 spec: config: '{ "cniVersion": "0.3.1", "type": "ip-preserving-host-device", "device": "fabric15" }' ``` Install the resources in the namespace where your RDMA workloads run: ```shell kubectl apply -f nads.yaml --namespace= ``` ## Attach the Fabric NICs to Pods Reference the `NetworkAttachmentDefinition` resources in the `k8s.v1.cni.cncf.io/networks` annotation of your Pod or Deployment manifest, and request the matching `rdma/fabric*` resources, as described in [Configure Multus CNI Plugin](https://docs.digitalocean.com/products/kubernetes/how-to/configure-multinode-gpus/index.html.md#configure-multus-cni-plugin) and [Manage RDMA-Related Resources](https://docs.digitalocean.com/products/kubernetes/how-to/configure-multinode-gpus/index.html.md#manage-rdma-related-resources). Include all 16 fabric NICs, `fabric0` through `fabric15`. For example, the annotation for all 16 NICs is: ```yaml metadata: annotations: k8s.v1.cni.cncf.io/networks: >- roce-net-fabric0@fabric0, roce-net-fabric1@fabric1, roce-net-fabric2@fabric2, roce-net-fabric3@fabric3, roce-net-fabric4@fabric4, roce-net-fabric5@fabric5, roce-net-fabric6@fabric6, roce-net-fabric7@fabric7, roce-net-fabric8@fabric8, roce-net-fabric9@fabric9, roce-net-fabric10@fabric10, roce-net-fabric11@fabric11, roce-net-fabric12@fabric12, roce-net-fabric13@fabric13, roce-net-fabric14@fabric14, roce-net-fabric15@fabric15 ``` When a Pod that references these resources starts, each fabric NIC it references moves from the node into the Pod, along with its addresses, routes, and VRF. The NICs no longer appear on the node until the Pod is deleted, when they return to the node and become available to other Pods. To confirm that a NIC kept its configuration, run `ip -br a` inside the Pod with `kubectl exec`. Each attached `fabric*` interface shows the same ULA and link-local address that it had on the node. ## Configure RDMA Applications Because each fabric NIC is bound to its own VRF, the fabric addresses are not reachable through the Pod’s default routing table. RDMA applications must either use the Pod’s primary interface, `eth0`, for connection establishment, or explicitly specify the source interface or VRF. For NCCL configuration on B300 worker nodes, see [Configure NCCL or RCCL](https://docs.digitalocean.com/products/kubernetes/how-to/configure-multinode-gpus/index.html.md#configure-nccl-or-rccl). When you choose a GID index, use the RoCE v2 entry for the NIC’s ULA, not its link-local address. To list the GID entries for the attached NICs, run `ibv_devinfo -v | grep GID` inside the Pod. For more information, see [GID Index Selection](https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-multi-node/index.html.md#gid-index-selection).