--- title: How to Configure Multi-Node B300 GPU Droplets on the Rail-Optimized Fabric description: Configure the RDMA network interfaces on multi-node NVIDIA B300 GPU Droplets in MKC1, which connect to a rail-optimized GPU fabric, and verify fabric … product: Droplets url: https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-rail-optimized-fabric/ last_updated: "2026-11-04" --- > **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document). # How to Configure Multi-Node B300 GPU Droplets on the Rail-Optimized Fabric DigitalOcean Droplets are Linux-based virtual machines (VMs) that run on top of virtualized hardware. Each Droplet you create is a new server you can use, either standalone or as part of a larger, cloud-based infrastructure. Multi-node NVIDIA B300 GPU Droplets with eight GPUs in the MKC1 datacenter connect to a rail-optimized GPU fabric. On this fabric, DigitalOcean assigns each Remote Direct Memory Access (RDMA) network interface controller (NIC) an IPv6 address and route, and you configure the NICs with a DigitalOcean-provided script instead of assigning addresses yourself. This page covers only the RDMA over Converged Ethernet (RoCE) network configuration of 8-GPU B300 Droplets in MKC1. For other multi-node GPU Droplets, and for the multi-node prerequisites and NCCL settings that apply to all of them, see [How to Configure Multi-Node GPU Droplets](https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-multi-node/index.html.md). ## Prerequisites - Multi-node GPU Droplets enabled for your account. To request them, [contact support](https://cloudsupport.digitalocean.com/s/). - [`doctl`](https://docs.digitalocean.com/reference/doctl/how-to/install/index.html.md) installed and authenticated, and `jq` and `curl` installed on your local machine, to create Droplets with the user data example. - A [personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/index.html.md) with permission to create Droplets. ## Fabric Network Planes The rail-optimized fabric is a Layer 3 routed network organized into two independent network planes. Each Droplet has 16 fabric NICs, eight on each plane: - **Plane 0**: The even-numbered interfaces, `eth2`, `eth4`, `eth6`, `eth8`, `eth10`, `eth12`, `eth14`, and `eth16`. - **Plane 1**: The odd-numbered interfaces, `eth3`, `eth5`, `eth7`, `eth9`, `eth11`, `eth13`, `eth15`, and `eth17`. Each GPU connects to one fabric NIC on each plane, with PCI affinity: | GPU | Plane 0 NIC | Plane 1 NIC | |---|---|---| | 0 | `eth2` | `eth3` | | 1 | `eth4` | `eth5` | | 2 | `eth6` | `eth7` | | 3 | `eth8` | `eth9` | | 4 | `eth10` | `eth11` | | 5 | `eth12` | `eth13` | | 6 | `eth14` | `eth15` | | 7 | `eth16` | `eth17` | Within a plane, any two Droplets in MKC1 can communicate across all rails. The two planes are isolated from each other, so a NIC on Plane 0 cannot reach a NIC on Plane 1. As on other multi-node GPU Droplets, `eth0` is for public internet connectivity and `eth1` is for private connectivity to other Droplets in the same VPC network. ## Configure the RDMA NICs DigitalOcean assigns each fabric NIC an IPv6 Unique Local Address (ULA), one `/64` per interface in the `fd02::/24` range, along with its routing information. A Droplet can read this configuration from the metadata service at `http://169.254.169.254/v1/gpu-fabric-nics`. The [`gen-netplan.sh`](https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-rail-optimized-fabric/gen-netplan.sh) script queries the metadata service, then writes a Netplan configuration to `/etc/netplan/90-rdma.yaml` and applies it. For each fabric NIC, the configuration sets the assigned IPv6 address and an MTU of 4200, and places the NIC in its own virtual routing and forwarding (VRF) instance with a dedicated routing table. Each routing table has a summary route to `fd02::/24` through the NIC’s leaf switch. This gives each NIC an isolated routing context for connectivity and path selection. The script requires `jq` and `curl` on the Droplet. You can customize it to fit your deployment workflow. For example, set the `SKIP_APPLY` environment variable to write the Netplan file without applying it, or `NETPLAN_OUT` to write it to a different path. The header of the script lists every supported override. ## View the gen-netplan.sh script ```bash #!/usr/bin/env bash # # Generate and apply a netplan config for RDMA devices. # # Usage: # gen-netplan.sh # # Script overrides: # METADATA_URL GPU fabric NICs endpoint # (default: http://169.254.169.254/v1/gpu-fabric-nics) # CONFIG_FILE skip metadata; read NIC JSON from this file instead # RDMA_DEV_FILE file containing `rdma dev` output (default: run `rdma dev`) # NETDEV_MAP file with " " lines (default: sysfs lookup) # RENDERER netplan renderer (default: networkd) # NETPLAN_OUT output path (default: /etc/netplan/90-rdma.yaml) # SKIP_APPLY if set, skip `netplan apply` set -euo pipefail METADATA_URL=${METADATA_URL:-http://169.254.169.254/v1/gpu-fabric-nics} RENDERER=${RENDERER:-networkd} MTU=${MTU:-4200} TABLE_BASE=${TABLE_BASE:-100} # first VRF table id; +1 per interface VRF_PREFIX=${VRF_PREFIX:-vrf-} # VRF name = (keep <=15 chars) SUMMARY=${SUMMARY:-fd02::/24} # on-link fabric summary route NETPLAN_OUT=${NETPLAN_OUT:-/etc/netplan/90-rdma.yaml} METADATA_RETRIES=${METADATA_RETRIES:-30} METADATA_RETRY_SECS=${METADATA_RETRY_SECS:-2} command -v jq >/dev/null 2>&1 || { echo "error: jq is required" >&2; exit 1; } command -v curl >/dev/null 2>&1 || { echo "error: curl is required" >&2; exit 1; } ensure_vrf_module() { if lsmod | awk '$1=="vrf" {found=1} END{exit !found}'; then echo "vrf module already loaded" return 0 fi } # Fetch NIC JSON from metadata (or CONFIG_FILE). Exits 1 on a permanent "not found". # Prints the config path on stdout when successful. fetch_config() { local config http_code body attempt config=$(mktemp) if [[ -n ${CONFIG_FILE:-} ]]; then [[ -f $CONFIG_FILE ]] || { echo "error: config '$CONFIG_FILE' not found" >&2; exit 1; } cp "$CONFIG_FILE" "$config" printf '%s' "$config" return 0 fi for attempt in $(seq 1 "$METADATA_RETRIES"); do body=$(mktemp) http_code=$(curl -sS -o "$body" -w '%{http_code}' --connect-timeout 2 --max-time 10 \ "$METADATA_URL" || true) case $http_code in 200) if ! jq -e . >/dev/null 2>&1 <"$body"; then echo "error: metadata returned invalid JSON" >&2 rm -f "$body" "$config" exit 1 fi mv "$body" "$config" printf '%s' "$config" return 0 ;; 404) # Transient while gpu-fabric.sh is still writing, or permanently absent. if grep -qi 'not found' "$body" 2>/dev/null || [[ ! -s $body ]]; then if [[ $attempt -lt $METADATA_RETRIES ]]; then echo "metadata: not found (attempt $attempt/$METADATA_RETRIES); retrying in ${METADATA_RETRY_SECS}s" >&2 rm -f "$body" sleep "$METADATA_RETRY_SECS" continue fi echo "error: GPU fabric NICs not found at $METADATA_URL" >&2 rm -f "$body" "$config" exit 1 fi echo "error: unexpected 404 body from $METADATA_URL: $(cat "$body")" >&2 rm -f "$body" "$config" exit 1 ;; 400) echo "error: hypervisor is not GPU fabric enabled (HTTP 400)" >&2 rm -f "$body" "$config" exit 1 ;; *) if [[ $attempt -lt $METADATA_RETRIES ]]; then echo "metadata: HTTP ${http_code:-000} (attempt $attempt/$METADATA_RETRIES); retrying in ${METADATA_RETRY_SECS}s" >&2 rm -f "$body" sleep "$METADATA_RETRY_SECS" continue fi echo "error: failed to fetch $METADATA_URL (HTTP ${http_code:-000})" >&2 rm -f "$body" "$config" exit 1 ;; esac done } get_rdma_dev() { if [[ -n ${RDMA_DEV_FILE:-} ]]; then cat "$RDMA_DEV_FILE"; else rdma dev; fi } # last 6 hex digits, lowercase, no separators (portable: no bash4-isms) last6() { local h=${1//:/} h=$(printf '%s' "$h" | tr 'A-F' 'a-f') printf '%s' "${h:${#h}-6}" } # netdev via sysfs (pipe-free; always returns 0, empty if not found) netdev_via_sysfs() { local dev=$1 p d p="/sys/class/infiniband/$dev/device/net" [[ -d $p ]] || return 0 for d in "$p"/*; do [[ -e $d ]] || break basename "$d" return 0 done } # netdev via `rdma link` (fallback; parses "... mlx5_X/port ... netdev ") netdev_via_rdmalink() { local dev=$1 rdma link 2>/dev/null | awk -v d="$dev" ' { split($2, a, "/"); if (a[1]==d) for (i=1;i<=NF;i++) if ($i=="netdev") { print $(i+1); exit } }' } # netdev name for an ib device (never aborts under set -e) netdev_for_dev() { local dev=$1 nd="" if [[ -n ${NETDEV_MAP:-} ]]; then nd=$(awk -v d="$dev" '$1==d {print $2; exit}' "$NETDEV_MAP") || nd="" else nd=$(netdev_via_sysfs "$dev") || nd="" [[ -n $nd ]] || { nd=$(netdev_via_rdmalink "$dev") || nd=""; } fi printf '%s' "$nd" } emit_netplan() { local config=$1 out=$2 local MAP ROWS matched idx MAP=$(mktemp) ROWS=$(mktemp) matched=0 while read -r _idx dev rest; do [[ -n $dev ]] || continue dev=${dev%:} guid=$(printf '%s' "$rest" | sed -n 's/.*node_guid \([0-9a-fA-F:]*\).*/\1/p') || guid="" [[ -n $guid ]] || continue nd=$(netdev_for_dev "$dev") || nd="" if [[ -z $nd ]]; then echo "warning: could not resolve a netdev for $dev; skipping" >&2 continue fi printf '%s %s\n' "$(last6 "$guid")" "$nd" >>"$MAP" matched=$((matched + 1)) done < <(get_rdma_dev) if [[ $matched -eq 0 ]]; then echo "warning: resolved 0 RDMA netdevs (check 'rdma dev' / sysfs / 'rdma link')" >&2 fi idx=0 while IFS=$'\t' read -r guid address prefix; do table=$((TABLE_BASE + idx)) idx=$((idx + 1)) key=$(last6 "$guid") nd=$(awk -v k="$key" '$1==k {print $2; exit}' "$MAP") if [[ -z $nd ]]; then echo "warning: no RDMA device matches guid $guid (last6 $key); skipping" >&2 continue fi # next hop = leaf side of the /64 (::2) via="${address%::*}::2" printf '%s\t%s\t%s\t%s\t%s\n' "$nd" "$address" "$prefix" "$table" "$via" >>"$ROWS" done < <(jq -r '.nics[] | [.guid, .address, .prefix] | @tsv' "$config") { printf 'network:\n version: 2\n renderer: %s\n' "$RENDERER" printf ' vrfs:\n' while IFS=$'\t' read -r nd address prefix table via; do printf ' %s%s:\n' "$VRF_PREFIX" "$nd" printf ' table: %s\n' "$table" printf ' interfaces:\n - %s\n' "$nd" done <"$ROWS" printf ' ethernets:\n' while IFS=$'\t' read -r nd address prefix table via; do printf ' %s:\n' "$nd" printf ' ignore-carrier: true\n' printf ' addresses:\n - %s%s\n' "$address" "$prefix" printf ' routes:\n - to: %s\n via: %s\n on-link: true\n' "$SUMMARY" "$via" printf ' mtu: %s\n' "$MTU" done <"$ROWS" } >"$out" rm -f "$MAP" "$ROWS" echo "wrote $out" } ensure_vrf_module CONFIG=$(fetch_config) if [[ -z $CONFIG ]]; then echo "error: no GPU fabric NIC configuration to apply" >&2 exit 1 fi trap 'rm -f "$CONFIG"' EXIT emit_netplan "$CONFIG" "$NETPLAN_OUT" if [[ -z ${SKIP_APPLY:-} ]]; then echo "applying netplan" netplan apply fi ``` Download the script to your local machine: ```shell curl -fsSLO https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-rail-optimized-fabric/gen-netplan.sh ``` You can run the script when you create a Droplet, using user data, or on a Droplet that is already running. ### With User Data Passing the script in the Droplet’s [user data](https://docs.digitalocean.com/products/droplets/how-to/provide-user-data/index.html.md) configures the fabric NICs on first boot. First, look up the IDs that the Droplet creation request needs. List the public images and find the NVIDIA AI/ML-ready image ID: ```shell doctl compute image list --public | grep NVIDIA ``` List your VPC networks and find the ID of the MKC1 VPC network to use: ```shell doctl vpcs list | grep mkc1 ``` List your SSH keys and note the IDs of the keys to add to the Droplet: ```shell doctl compute ssh-key list ``` Export your personal access token: ```shell export DIGITALOCEAN_ACCESS_TOKEN= ``` From the directory that contains `gen-netplan.sh`, run the following command to create the Droplet. Replace `` with the multi-node B300 plan slug that support enabled for your account, and replace ``, ``, and `` with the values from the previous commands: ```shell jq -n \ --arg name "rail-optimized-demo" \ --arg region "mkc1" \ --arg size "" \ --argjson image \ --arg vpc_uuid "" \ --argjson ssh_keys '[]' \ --rawfile script ./gen-netplan.sh \ '{ name:$name, region:$region, size:$size, image:$image, vpc_uuid:$vpc_uuid, ssh_keys:$ssh_keys, user_data: ( "#cloud-config\n" + "package_update: true\n" + "packages:\n - jq\n - curl\n" + "write_files:\n" + "- path: /usr/local/bin/gen-netplan.sh\n" + " encoding: b64\n" + " permissions: \"0755\"\n" + " content: " + ($script|@base64) + "\n" + "runcmd:\n" + " - [/usr/local/bin/gen-netplan.sh]\n" ) }' | curl -ifsS -X POST "https://api.digitalocean.com/v2/droplets" \ -H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" \ -H "Content-Type: application/json" \ -d @- ``` The user data installs `jq` and `curl`, writes the script to `/usr/local/bin/gen-netplan.sh`, and runs it once on first boot. The Netplan configuration it writes persists across reboots. ### On a Running Droplet To configure a Droplet that is already running, copy the script to the Droplet. Replace `` with the Droplet’s public IP address: ```shell scp gen-netplan.sh root@:/root/ ``` Then install the script’s dependencies and run it: ```shell ssh root@ 'apt-get update -y && apt-get install -y jq curl && chmod +x /root/gen-netplan.sh && /root/gen-netplan.sh' ``` ## Verify the NIC Configuration After the script runs, [connect to the Droplet](https://docs.digitalocean.com/products/droplets/how-to/connect-with-ssh/index.html.md) and list the network interfaces: ```shell ip -br a ``` Every fabric interface on both planes must be `UP` and have two IPv6 addresses: the ULA from the metadata service, and an automatically assigned link-local address in `fe80::/10`. The link-local address is not routable; do not use it for RDMA. The second 16-bit group of the ULA identifies the plane: `fd02:0:...` for Plane 0 and `fd02:1:...` for Plane 1. A VRF device for each interface, `vrf-eth2` through `vrf-eth17`, must also exist and be `UP`. Your addresses differ from the following example: ```text Output eth0 UP 203.0.113.85/20 10.52.0.8/16 fe80::a077:4bff:fe76:1835/64 eth1 UP 10.132.0.37/20 fe80::f810:96ff:fe8d:4ad6/64 eth2 UP fd02:0:400:527::1/64 fe80::ac3a:e2ff:fecf:d75e/64 eth3 UP fd02:1:400:527::1/64 fe80::ac3a:e2ff:fecf:d75f/64 eth4 UP fd02:0:500:527::1/64 fe80::ac3a:e2ff:fecf:d77e/64 eth5 UP fd02:1:500:527::1/64 fe80::ac3a:e2ff:fecf:d77f/64 eth6 UP fd02:0:600:527::1/64 fe80::ac3a:e2ff:fecf:d78e/64 eth7 UP fd02:1:600:527::1/64 fe80::ac3a:e2ff:fecf:d78f/64 eth8 UP fd02:0:700:527::1/64 fe80::ac3a:e2ff:fecf:d76e/64 eth9 UP fd02:1:700:527::1/64 fe80::ac3a:e2ff:fecf:d76f/64 eth10 UP fd02:0:300:527::1/64 fe80::ac3a:e2ff:fecf:d7ce/64 eth11 UP fd02:1:300:527::1/64 fe80::ac3a:e2ff:fecf:d7cf/64 eth12 UP fd02:0:200:527::1/64 fe80::ac3a:e2ff:fecf:d7ae/64 eth13 UP fd02:1:200:527::1/64 fe80::ac3a:e2ff:fecf:d7af/64 eth14 UP fd02:0:100:527::1/64 fe80::ac3a:e2ff:fecf:d79e/64 eth15 UP fd02:1:100:527::1/64 fe80::ac3a:e2ff:fecf:d79f/64 eth16 UP fd02:0:0:527::1/64 fe80::ac3a:e2ff:fecf:d7be/64 eth17 UP fd02:1:0:527::1/64 fe80::ac3a:e2ff:fecf:d7bf/64 ... vrf-eth10 UP vrf-eth11 UP vrf-eth12 UP vrf-eth13 UP vrf-eth14 UP vrf-eth15 UP vrf-eth16 UP vrf-eth17 UP vrf-eth2 UP vrf-eth3 UP vrf-eth4 UP vrf-eth5 UP vrf-eth6 UP vrf-eth7 UP vrf-eth8 UP vrf-eth9 UP ``` The routing table of each VRF contains the directly connected `/64` route for its fabric interface and a summary route to `fd02::/24` through the leaf switch, which uses the `::2` address on the same `/64`. To check the routing table for `eth2`, run: ```shell ip -6 route show vrf vrf-eth2 ``` The output looks similar to the following: ```text Output fd02:0:400:527::/64 dev eth2 proto kernel metric 256 pref medium fd02::/24 via fd02:0:400:527::2 dev eth2 proto static metric 1024 onlink pref medium fe80::/64 dev eth2 proto kernel metric 256 pref medium multicast ff00::/8 dev eth2 proto kernel metric 256 pref medium ``` ## Verify Fabric Connectivity First, verify that every fabric NIC can reach its gateway, the leaf switch at the `::2` address on the NIC’s `/64`. Run the ping from inside the NIC’s VRF. For example, for `eth2`: ```shell ip vrf exec vrf-eth2 ping -c3 fd02:0:400:527::2 ``` To check all 16 fabric NICs at once, run the following loop. All 16 pings must succeed: ```shell for i in $(seq 2 17); do gw=$(ip -6 route show vrf vrf-eth$i | awk '/via/ {print $3; exit}') ip vrf exec vrf-eth$i ping -c3 "$gw" done ``` Next, verify connectivity across rails and Droplets within each plane. From each fabric NIC on this Droplet, ping the ULA (the `::1` address) of each of the eight fabric NICs on the same plane of a peer Droplet, for 64 pings per plane. All in-plane pings must succeed. For example, replace `` and `` with addresses from the peer Droplet’s `ip -br a` output: ```shell ip vrf exec vrf-eth2 ping -c3 ip vrf exec vrf-eth17 ping -c3 ``` Pings between the two planes, such as from `eth2` to a peer’s `eth3`, fail because the planes are isolated. ## Configure RDMA Applications Because each fabric NIC is bound to its own VRF, the fabric addresses are not reachable through the default routing table. A command that doesn’t specify a source interface or VRF, such as `ping fd02:0:400:527::1`, fails with `Network is unreachable`. RDMA applications must either use the out-of-band interface, `eth1`, for connection establishment, or explicitly specify the source interface or VRF. For NCCL configuration on B300 Droplets, see [NVIDIA B300](https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-multi-node/index.html.md#nvidia-b300). When you choose a GID index, use the RoCE v2 entry for the NIC’s ULA, not its link-local address. See [GID Index Selection](https://docs.digitalocean.com/products/droplets/how-to/gpu/configure-multi-node/index.html.md#gid-index-selection).