For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

DigitalOcean Kubernetes (DOKS) can automatically replace worker nodes that stay unhealthy. The Data Plane Operator carries out remediation. It runs in your cluster, watches Node objects, and applies rules you define through cluster-scoped custom resources.

Remediation is opt-in. You enable it for the cluster, define rules for which node conditions trigger a replacement, and monitor remediation through Kubernetes events.

How Automatic Node Remediation Works

When remediation is enabled, the Data Plane Operator watches each worker Node object and, for each one:

1. Checks whether remediation is enabled cluster-wide in the DataPlaneOperatorConfig object.
2. Selects the NodeRemediationConfig (NRC) whose nodeSelector best matches the node.
3. Evaluates the NRC’s spec.unhealthyConditions against the node’s status.conditions.
4. After the configured grace and duration thresholds pass, issues a remediation action that replaces the worker through the DigitalOcean API.

The operator does not decide what counts as unhealthy for your applications. It reacts to node conditions on each worker: standard kubelet conditions (for example, Ready=False) and custom condition types you define for your environment. You, or software you run in the cluster, set those conditions on the node. The operator replaces the worker when a condition matches your NodeRemediationConfig rules for long enough.

You can set custom conditions with tools such as Node Problem Detector, a DaemonSet or operator that patches node status, or automation tied to your own health checks. The condition type and status strings in your rules must match exactly what you publish on the node.

%%{init: {"themeVariables": {"edgeLabelBackground": "transparent"}} }%%
sequenceDiagram
    participant User as Your CR configuration
    participant DPO as Data Plane Operator
    participant Node as Node (status.conditions)
    participant API as DigitalOcean API
    User ->> DPO: DataPlaneOperatorConfig (enabled)
    User ->> DPO: NodeRemediationConfig (rules)
    Node ->> DPO: Condition updates
    DPO ->> DPO: Match NRC, check duration and budget
    DPO ->> Node: Remediation annotations
    DPO ->> API: cordon-drain-replace or force-replace
    DPO ->> User: Events on NodeRemediationConfig

Prerequisites

Before you configure remediation, make sure you have:

• kubectl access to the cluster, with permission to create cluster-scoped resources in the dataplane-operator.doks.digitalocean.com API group.
• Worker nodes that carry the doks.digitalocean.com/node-pool-id label. The operator requires this label to scope replacements to the correct node pool and to observe healthy replacements. DOKS applies it automatically. A node without this label is not remediated.
• Workloads that tolerate node loss. Remediation replaces the underlying Droplet, so use PodDisruptionBudgets, multiple replicas, and similar safeguards.

API Group and Resources

You configure remediation with two cluster-scoped custom resources in the dataplane-operator.doks.digitalocean.com/v1alpha1 API group:

In kubectl, refer to them by these names:

• dataplaneoperatorconfig / dataplaneoperatorconfigs
• noderemediationconfig / noderemediationconfigs

Apply or update these resources with kubectl apply or your GitOps workflow. The operator reads the live objects from the Kubernetes API. Changes to a NodeRemediationConfig take effect on the next node reconcile.

You can add NodeRemediationConfig objects for different node pools or custom health signals.

Enable Remediation for the Cluster

Remediation is gated for the entire cluster by a single DataPlaneOperatorConfig object named user-config. The operator reads only the object with this exact name and ignores any other name. If user-config does not exist, remediation is disabled until you create the object and set spec.nodeConditionRemediation.enabled to true.

To enable remediation, create the object:

apiVersion: dataplane-operator.doks.digitalocean.com/v1alpha1
kind: DataPlaneOperatorConfig
metadata:
  name: user-config
spec:
  nodeConditionRemediation:
    enabled: true

The spec.nodeConditionRemediation block is optional. When it is absent, remediation behaves as disabled (the same as enabled: false). When it is present, enabled is required:

Per-policy limits, such as inFlightLimit, observationDeadline, and the unhealthy rules, are set on NodeRemediationConfig, not on this object.

To verify that remediation is enabled, check the object:

kubectl get dataplaneoperatorconfig user-config -o yaml

Even when remediation is disabled, the operator can still export node condition metrics. It issues no replacements until remediation is enabled.

Define Remediation Rules

A NodeRemediationConfig (NRC) is a cluster-scoped policy that defines which node conditions trigger remediation, how long to wait, and what action to take. You can define multiple NRCs. The operator selects one per node, as described in Node Selector Matching.

An NRC supports the following top-level spec fields:

Unhealthy Condition Rules

Each entry in unhealthyConditions has the following shape:

remediationAction accepts the following values:

There is no alert-only action. Both supported values replace the worker.

The operator checks rules in list order. The first matching unhealthy condition whose duration has elapsed drives remediation.

Built-In Node Conditions

The following are the standard Kubernetes node conditions. Pair the type and status to match your monitoring or alerting model:

Custom Node Conditions

Kubernetes allows any condition type string on Node.status.conditions. Use custom types when the built-in signals (Ready, DiskPressure, and so on) are not enough, for example, application-specific health, GPU drivers, or checks only your team understands.

The operator does not create custom conditions for you. You choose the type name, publish the condition on unhealthy nodes, and reference the same type and status in unhealthyConditions. The operator then:

• Compares each configured (type, status) pair to the node’s current conditions.
• Waits for the configured duration after lastTransitionTime.
• Treats a new worker as healthy enough to complete observation when it is Ready=True and does not match any unhealthyConditions entry on the selected NRC.

To add a custom condition:

1. Choose a unique type string for your organization or workload, for example CustomNodeCondition.
2. Set that condition on nodes your health system considers failed. The status is usually "True".
3. Add a matching entry to your NodeRemediationConfig with a duration and a remediationAction.
4. Confirm the condition appears on a test node with kubectl describe node <node-name>, under Conditions.

The following NRC fragment matches a custom condition:

unhealthyConditions:
  - type: CustomNodeCondition
    status: "True"
    duration: 10m
    remediationAction: cordon-drain-replace

Use the same type and status strings everywhere: in the detector, the NRC, and your runbooks.

Node Selector Matching

Only one NodeRemediationConfig applies to a given node at a time. When more than one NRC could apply to a node based on its nodeSelector, the operator picks the best match using this precedence:

1. The config with the largest number of matching selector keys wins.
2. If no selector matches, the operator uses an NRC with an empty nodeSelector ({} or omitted) as the default, if one exists.
3. If no selector matches and there is no default, the node is not remediated under any NRC.

In the following example, a pool-specific config overrides the default:

# Default for all nodes
apiVersion: dataplane-operator.doks.digitalocean.com/v1alpha1
kind: NodeRemediationConfig
metadata:
  name: default
spec:
  startupGracePeriod: 15m
  inFlightLimit: 3
  observationDeadline: 4h
  nodeSelector: {}
  unhealthyConditions:
    - type: Ready
      status: "False"
      duration: 10m
      remediationAction: cordon-drain-replace
---
# Stricter policy for a GPU pool
apiVersion: dataplane-operator.doks.digitalocean.com/v1alpha1
kind: NodeRemediationConfig
metadata:
  name: gpu-pool
spec:
  startupGracePeriod: 20m
  inFlightLimit: 1
  observationDeadline: 4h
  nodeSelector:
    doks.digitalocean.com/node-pool: gpu-workers
  unhealthyConditions:
    - type: CustomGPUHealthSignal
      status: "True"
      duration: 5m
      remediationAction: cordon-drain-replace

We recommend a longer startup grace period for large GPU nodes, which can take longer to initialize.

Disable Remediation for a Subset of Nodes

To disable remediation for a group of nodes, set inFlightLimit: 0 on the NRC that those nodes select. A cluster-wide enabled: true does not override this.

Remediation Lifecycle

Preconditions

The operator remediates a node only when all of the following hold:

1. DataPlaneOperatorConfig/user-config has spec.nodeConditionRemediation.enabled: true.
2. A matching NRC exists with an inFlightLimit greater than 0.
3. The node is past its startupGracePeriod, measured from creation.
4. An unhealthyConditions rule matches and its duration has elapsed since lastTransitionTime.
5. The node is not already marked with a remediation-issued annotation.
6. The node has the doks.digitalocean.com/node-pool-id label, which is required for pool-scoped replacement and observation.

Remediation Steps

When the preconditions hold, the operator:

1. Reserves an in-flight slot on the matched NRC.
2. Sets remediation annotations on the node.
3. Calls the DigitalOcean API to replace the worker (cordon-drain-replace or force-replace).
4. Emits a RemediationIssued event on the NRC.
5. Releases the slot and emits a RemediationObserved event when a healthy replacement appears in the same node pool.

In-Flight Remediation Budget

Each NRC has its own cap, set by inFlightLimit. A remediation is in flight from the moment it is issued until one of the following happens:

• A healthy replacement is observed in the same doks.digitalocean.com/node-pool-id (a RemediationObserved event).
• The observationDeadline elapses without observation (a RemediationUnobserved event).

When the budget is full, the operator does not issue another remediation for that NRC. It rechecks about once a minute and starts a new remediation when a slot frees up.

Monitor Remediation with Events

The operator records Kubernetes events on the NodeRemediationConfig that applied to the node, not on the Node object. Events are the primary way to see what happened during remediation.

To view events for an NRC:

kubectl describe noderemediationconfig <nrc-name>

To list remediation events across all NRCs:

kubectl get events --field-selector involvedObject.kind=NodeRemediationConfig

The operator emits the following events:

A RemediationIssued message looks like this:

Issued cordon-drain-replace on node "worker-pool-abc123" (pool "uuid-...").

A RemediationObserved message looks like this:

Observed healthy replacement for node "worker-pool-abc123" in pool "uuid-..." (action cordon-drain-replace issued at 2026-05-29T12:00:00Z).

Full Configuration Example

The following manifest enables remediation for the cluster and defines a default NRC that replaces nodes reporting a custom unhealthy signal:

apiVersion: dataplane-operator.doks.digitalocean.com/v1alpha1
kind: DataPlaneOperatorConfig
metadata:
  name: user-config
spec:
  nodeConditionRemediation:
    enabled: true
---
apiVersion: dataplane-operator.doks.digitalocean.com/v1alpha1
kind: NodeRemediationConfig
metadata:
  name: default
spec:
  startupGracePeriod: 10m
  inFlightLimit: 3
  observationDeadline: 4h
  nodeSelector: {}
  unhealthyConditions:
    - type: CustomNodeSignal
      status: "True"
      duration: 10m
      remediationAction: cordon-drain-replace

Troubleshooting