How to Use the Routing Agent in Kubernetes Clusterspublic

The routing agent is a Kubernetes controller that manages IP routes on Kubernetes worker nodes. The controller is deployed as a DaemonSet and is available to DOKS customers at no additional cost. Using the routing agent, you can:

• Manage IP routes using custom resources

• Define multiple gateways with automatic Equal Cost Multi Path (ECMP) setup

• Override default routes

• Enable precise control over network configurations by applying routes to specific nodes using label selectors

These features are especially useful for setting up VPN and routing outbound traffic through self-managed VPC gateways. For examples of various route definitions, see Define Routes.

You can enable or disable the routing agent only using the DigitalOcean CLI or API.

Enable the Routing Agent Using the DigitalOcean CLI

To enable the routing agent when creating a cluster, set the --enable-routing-agent flag to true.

The following example creates a cluster named example-cluster in the nyc1 region with one node pool using the latest Kubernetes version and the routing agent enabled.

doctl kubernetes cluster create example-cluster --region nyc1 --version latest --enable-routing-agent=true

To enable the agent for an existing cluster, update the cluster with the --enable-routing-agent flag to true. For example:

doctl kubernetes cluster update example-cluster --enable-routing-agent=true

Enable the Routing Agent Using the DigitalOcean API

Enable the routing agent for a cluster using the DigitalOcean API with cURL or Go.

curl --location 'https://api.digitalocean.com/v2/kubernetes/clusters' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DIGITALOCEAN_TOKEN' \
--data '{
    "name": "example-cluster",
    "region": "syd1",
    "version": "1.32.2-do.0",
    "node_pools": [
        {
            "size": "s-1vcpu-2gb",
            "count": 3,
            "name": "worker-pool"
        }
    ],
    "routing_agent": {
        "enabled": true
    }
}'

curl --location --request PUT 'https://api.digitalocean.com/v2/kubernetes/clusters/{cluster_id}' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DIGITALOCEAN_TOKEN' \
--data '{
    "name": "example-cluster",
    "routing_agent": {
        "enabled": true
    }
}'

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/digitalocean/godo"
)

func main() {
	client := godo.NewFromToken("your-digitalocean-token")

	cluster, _, err := client.Kubernetes.Create(context.Background(), &godo.KubernetesClusterCreateRequest{
		Name:        "example-cluster",
		RegionSlug:  "nyc1",
		VersionSlug: "1.32.2-do.0",
		NodePools: []*godo.KubernetesNodePoolCreateRequest{
			{
				Name:  "worker-pool",
				Count: 3,
				Size:  "s-1vcpu-2gb",
			},
		},
		RoutingAgent: &godo.KubernetesRoutingAgent{
			Enabled: godo.PtrTo(true),
		},
	})
	if err != nil {
		fmt.Printf("Error creating cluster: %s\n", err)
		os.Exit(1)
	}

	isEnabled := *cluster.RoutingAgent.Enabled
	fmt.Printf("Cluster creation successfully issued with routing agent enabled=%v\n", isEnabled)
}

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/digitalocean/godo"
)

func main() {
	client := godo.NewFromToken("your-digitalocean-token")

	cluster, _, err := client.Kubernetes.Update(context.Background(), "your-cluster-id", &godo.KubernetesClusterUpdateRequest{
		Name: "example-cluster",
		RoutingAgent: &godo.KubernetesRoutingAgent{
			Enabled: godo.PtrTo(true),
		},
	})

	if err != nil {
		fmt.Printf("Error updating cluster: %s\n", err)
		os.Exit(1)
	}

	isEnabled := *cluster.RoutingAgent.Enabled
	fmt.Printf("Cluster update successfully issued with routing agent enabled=%v\n", isEnabled)
}

Define Routes

When the routing agent is enabled on your cluster, you can define routes for the worker nodes by adding custom resources of kind: Route. The agent can add routes to a node only after the Linux kernel validates conditions such as the gateway IP being reachable.

The following example defines a route named basic that routes traffic to the destination nodes IP ranges 1.2.3.4/5 through a gateway IP address 10.114.0.3.

apiVersion: networking.doks.digitalocean.com/v1alpha1
kind: Route
metadata:
  name: basic
spec:
  destinations:
    - "1.2.3.4/5"
  gateways:
    - "10.114.0.3" 

Define Multiple Gateways

The following route configuration defines multiple gateways.

apiVersion: networking.doks.digitalocean.com/v1alpha1
kind: Route
metadata:
  name: basic
spec:
  destinations:
    - "1.2.3.4/5"
  gateways:
    - "10.114.0.3"
    - "10.114.0.4"

When you specify multiple gateways, an IP route with multiple hops is created and Equal Cost Multi Path (ECMP) is applied to those routes. ECMP selects which gateway to route traffic to by computing a hash based on the source and destination IP addresses and port. Thus, traffic is split over multiple gateways, allowing for larger traffic volumes.

Allowing ICMP on gateways is required for routes with multiple gateways. This is because the routing agent sends ICMP probes (pings) to the gateways at regular intervals of 30 seconds and the Linux kernel uses the ICMP echo reply. If one of the gateways becomes unreachable and fails to establish network flow repeatedly, no traffic is sent to that gateway anymore. When the ping succeeds, the previously-failed gateway is put back into the rotation once it recovers and sending traffic through it starts again.

Override Default Route

The routing agent lets you override the default route without impacting the overall cluster connectivity. For example, the following route defines that to override the default route and send all packets to 10.114.0.3 as the next hop.

apiVersion: networking.doks.digitalocean.com/v1alpha1
kind: Route
metadata:
  name: basic
spec:
  destinations:
    - "0.0.0.0/0" # default route on Linux
  gateways:
    - "10.114.0.3"
    - "10.114.0.4"

To ensure the connectivity between Kubernetes worker nodes and the Kubernetes API server, and prevent problems when overriding the default route, the routing agent creates /32 routes for the control plane endpoint using the Droplet’s default gateway.

Select Node Routes

You can select routes to create on specific nodes using the nodeSelector field that matches the individual node labels. The nodeSelector is of core/v1/NodeSelector type. For example:

apiVersion: networking.doks.digitalocean.com/v1alpha1
kind: Route
metadata:
  name: basic
spec:
  destinations:
    - "1.2.3.4/5" 
  gateways:
    - "10.114.0.3"
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
      - key: doks.digitalocean.com/node-pool
        operator: In
        values: ["worker-pool"]

The agent supports the following operators to match on node labels:

• In: Value of label key needs to match any of values

• NotIn: Value of label key must not match any of values

• Exists: Label key must exist on node

• DoesNotExist: Label key must not exist on node

Alternatively, you can directly select a node by its name using the matchFields field:

apiVersion: networking.doks.digitalocean.com/v1alpha1
kind: Route
metadata:
  name: basic
spec:
  destinations:
    - "1.2.3.4/5" # configures nets to be routed via GW
  gateways:
    - "10.114.0.3" # gateway IP
  nodeSelector:
    nodeSelectorTerms:
    - matchFields:
      - key: metadata.name
        operator: In
        values: ["worker-pool-y"]

VPC Gateway Droplet

Using a self-managed VPC gateway Droplet along with the routing agent, you can configure static egress IP addresses and ensure that the outbound traffic from Kubernetes workloads always originate from a predictable IP address.

Custom Routing Table and Rules

The routing agent creates a custom routing table and rule for the public IP address of the Droplet. The routing tables and rules are used to ensure symmetric routing on eth0 so that packets that enter via eth0 also leave via eth0. Symmetric routing is required for overriding the default route when using network load balancers.

Users must ensure that there is no overlap with any custom modifications on the routing table names and id as well as the ip rule priority.

• Custom Routing Table Name: eth0table
• Custom Routing Table Number: 135
• IP Rule Priority for Custom Routing Table: 15135

Remove the Routing Agent

To remove the agent, we recommend that you first remove the route CRDs. Deleting routes is protected by a finalizer on the agent and can take up to 15 seconds before succeeding.

Then, disable the agent using the CLI or API.

doctl kubernetes cluster update example-cluster --enable-routing-agent=false

curl --location --request PUT 'https://api.digitalocean.com/v2/kubernetes/clusters/{cluster_id}' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $DIGITALOCEAN_TOKEN' \
--data '{
    "name": "example-cluster",
    "routing_agent": {
        "enabled": false
    }
}'

Disabling the agent stops reconciling the routing-agent DaemonSet. However, it doesn’t remove the DaemonSet from the cluster. You should remove the DaemonSet and the associated resources by removing all resources with the label c3.doks.digitalocean.com/component=routing-agent. For example:

kubectl delete -n kube-system --selector c3.doks.digitalocean.com/component=routing-agent CustomResourceDefinition,ServiceAccount,Role,ClusterRole,RoleBinding,ClusterRoleBinding,DaemonSet