How to Configure Multi-Node GPU Droplets

To use our user data script, you must adopt a specific naming convention for your Droplets:

• The name must end with a hyphen, -, followed by an integer between 1 to 254. For example, examplename-1.
• The name must have no other hyphens.

Then, use the following cloud-config file when you create the Droplet:

#cloud-config
write_files:
- path: /usr/sbin/gpu-fabric.sh
  content: |
    #!/bin/bash
    IFACES=$(ip -br addr | grep eth | grep -E 'eth2|eth3|eth4|eth5|eth6|eth7|eth8|eth9' | awk '{print $1}')
    subnet=50
    octet=$(hostname | cut -d '-' -f 2)
    for i in ${IFACES}; do
        /usr/sbin/ip link set dev ${i} up
        /usr/sbin/ip link set dev ${i} mtu 4200
        ADDR="192.168.${subnet}.${octet}/24"
        /usr/sbin/ip addr add dev ${i} ${ADDR}
        subnet=$((subnet + 1))
    done
    /usr/sbin/ip -br addr
  permissions: '0755'
bootcmd:
- /usr/sbin/gpu-fabric.sh
runcmd:
- /usr/sbin/gpu-fabric.sh

You can pass this script when creating a GPU Droplet with doctl by using the -user-data-file flag.

You can use Netplan to configure the NICs. The AI/ML-ready image we provide for GPU Droplets includes Netplan support.

On each Droplet, open /etc/netplan/50-cloud-init.yaml and add the following block after eth1:

eth2:
  dhcp4: false
  dhcp6: false
  link-local: []          
  addresses:
    - 192.168.50.2/24
  mtu: 4200
eth3:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.51.2/24
  mtu: 4200
eth4:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.52.2/24
  mtu: 4200
eth5:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.53.2/24
  mtu: 4200
eth6:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.54.2/24
  mtu: 4200
eth7:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.55.2/24
  mtu: 4200 
eth8:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.56.2/24
  mtu: 4200
eth9:
  dhcp4: false
  dhcp6: false
  link-local: []
  addresses:
    - 192.168.57.2/24
  mtu: 4200

You can optionally also edit the eth1 MTU to 9002.

Save the file and apply the changes:

sudo netplan apply

Repeat this process on every other Droplet, replacing the fourth octet each time. For example, change 192.168.50.2 to 192.168.50.3 on the next Droplet, then to 192.168.50.4 on the next, and so on.

You can use our gpu-fabric Ansible playbook to configure multi-node GPU Droplets:

DigitalOcean's gpu-fabric GitHub Repository

A simple Ansible playbook to configure multi-node GPU Droplets.

The README of the repository has installation and usage instructions which are replicated here:

This content is automatically generated from https://github.com/digitalocean/gpu-fabric/blob/main/README.md.

This repository contains a simple Ansible playbook to configure multi-node GPU Droplets.

To use this playbook:

1. On the machine that you will use to run this playbook, first install Ansible and then clone this repository.

2. In the inventory/droplets file in your cloned version of this repository, in the [multinode_gpu_droplets] section, specify the public IP addresses of your GPU Droplets.

3. Ansible uses SSH under the hood to configure Droplets. If you have never connected to your Droplets with SSH and the .ssh/config file on your machine does not include StrictHostKeyChecking no, add the following line to the inventory/droplets file:

ansible_ssh_common_args='-o StrictHostKeyChecking=no'

4. Save the file, then run the playbook from the root of the repository:

ansible-playbook -i inventory/droplets customer-play.yaml

The output of a successful run looks similar to the following:

PLAY [multinode_gpu_droplets] ***********************************************************************************

TASK [Gathering Facts] ******************************************************************************************
ok: [10.10.10.10]

TASK [read /etc/netplan/50-cloud-init.yaml] *********************************************************************
ok: [10.10.10.10]

TASK [extract /etc/netplan/50-cloud-init.yaml] ******************************************************************
ok: [10.10.10.10]

TASK [set a unique index for each droplet] **********************************************************************
ok: [10.10.10.10] => (item=10.10.10.10)

TASK [adjust /etc/netplan/50-cloud-init.yaml] *******************************************************************
ok: [10.10.10.10]

TASK [write /etc/netplan/50-cloud-init.yaml] ********************************************************************
ok: [10.10.10.10]

TASK [install lldp] *********************************************************************************************
ok: [10.10.10.10]

PLAY RECAP ******************************************************************************************************
10.10.10.10             : ok=7    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0