---
title: How to Use the Node Readiness Controller in DOKS
description: DOKS uses a node readiness controller to gate pod scheduling until all required node components, including GPU drivers, are healthy.
product: Kubernetes
url: https://docs.digitalocean.com/products/kubernetes/how-to/use-node-readiness-controller/
last_updated: "2026-08-12"
---

> **For AI agents:** The documentation index is at [https://docs.digitalocean.com/llms.txt](https://docs.digitalocean.com/llms.txt). Markdown versions of pages use the same URL with `index.html.md` in place of the HTML page (for example, append `index.html.md` to the directory path instead of opening the HTML document).

# How to Use the Node Readiness Controller in DOKS

DigitalOcean Kubernetes (DOKS) is a Kubernetes service with a fully managed control plane, high availability, and autoscaling. DOKS integrates with standard Kubernetes toolchains and DigitalOcean’s load balancers, volumes, CPU and GPU Droplets, API, and CLI.

DigitalOcean Kubernetes (DOKS) runs a node readiness controller as a managed component that prevents pods from being scheduled onto nodes until all required components are healthy. The controller monitors custom node conditions and manages `NoSchedule` taints automatically, removing them once the associated conditions are met.

**Warning**:

  Do not modify the node readiness controller or its `NodeReadinessRule` resources. These are managed by DigitalOcean and any changes you make are overwritten by the reconciler. You can still customize GPU health metrics through a supported ConfigMap, as described in [Override GPU Health Metrics](#override-gpu-health-metrics). For more information about managed components, see [The Managed Elements of DigitalOcean Kubernetes](https://docs.digitalocean.com/products/kubernetes/details/managed/index.html.md).

## How the Node Readiness Controller Works

The controller is built on the [upstream `node-readiness-controller` project](https://github.com/kubernetes-sigs/node-readiness-controller). It introduces a `NodeReadinessRule` custom resource that declares:

- **Conditions:** One or more node conditions that must all have the required status.
- **Taint:** The `NoSchedule` taint to apply to the node while conditions are unmet.
- **Node selector:** Which nodes the rule targets, using label selectors.
- **Enforcement mode:** `bootstrap-only` evaluates the rule only during node startup; `continuous` re-evaluates it throughout the node’s lifetime.

When a node does not satisfy all conditions in a rule, the controller applies the rule’s taint to the node. When all conditions are satisfied, the controller removes the taint and the node becomes available for scheduling.

The controller uses only taints with keys prefixed by `readiness.k8s.io/`. These taints are distinct from the GPU device plugin taints (`amd.com/gpu:NoSchedule`, `nvidia.com/gpu:NoSchedule`) described in [Automatic Application of Labels and Taints to Nodes](https://docs.digitalocean.com/products/kubernetes/details/managed/index.html.md#automatic-application-of-labels-and-taints-to-nodes).

## Managed Node Readiness Rules

DOKS deploys three `NodeReadinessRule` resources, all using `continuous` mode so they re-evaluate conditions throughout the node lifecycle, not only during bootstrap. The controller and the `NodeReadinessRule` custom resource definition are present on all clusters, but the three managed rules require Kubernetes `1.33.9-do.6`, `1.34.5-do.6`, `1.35.1-do.6`, `1.36.0-do.0`, or later. Clusters on earlier patch releases run the controller without the managed rules until you upgrade.

| Rule | Targeted nodes | Required conditions | Managed taint | Dry run |
|---|---|---|---|---|
| `doks-critical` | All worker nodes (non-control-plane) | `doks.kubernetes.io/CiliumAgentReady: True`, `doks.kubernetes.io/CPBridgeReady: True` | `readiness.k8s.io/DOKSCriticalComponentsReady:NoSchedule` | False |
| `amd-gpu-critical` | Nodes labeled `doks.digitalocean.com/gpu-brand: amd` | `doks.kubernetes.io/AmdGpuReady: True` | `readiness.k8s.io/DOKSAmdGpuReady:NoSchedule` | True (Public Preview) |
| `nvidia-gpu-critical` | Nodes labeled `doks.digitalocean.com/gpu-brand: nvidia` with the Data Center GPU Manager (DCGM) enabled (`doks.digitalocean.com/nvidia-dcgm-enabled: "true"`) | `doks.kubernetes.io/NvidiaGpuReady: True` | `readiness.k8s.io/DOKSNvidiaGpuReady:NoSchedule` | True (Public Preview) |

DOKS sets the `doks.kubernetes.io/CiliumAgentReady` and `doks.kubernetes.io/CPBridgeReady` conditions from the health of the Cilium agent and the control plane bridge (cpbridge) running on each node. The GPU conditions are set by a separate reporter, as described in the next section.

## GPU Node Health Conditions

DOKS runs a GPU node condition reporter as a DaemonSet on GPU worker nodes. It sets the `doks.kubernetes.io/AmdGpuReady` or `doks.kubernetes.io/NvidiaGpuReady` node condition by querying the metrics endpoint of the GPU metrics exporter running on the same node.

The reporter evaluates the condition as follows:

- If the metrics endpoint is reachable and returns a 2xx status code, and none of the tracked failure metrics have a value greater than zero, the condition is set to `True`.
- If the endpoint is unreachable, returns a non-2xx status code, or any tracked failure metric has a value greater than zero, the condition is set to `False`.

The reporter checks the following metrics:

**NVIDIA** (reported by the [DCGM Exporter](https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html)):

- `DCGM_FI_DEV_XID_ERRORS` and `DCGM_FI_DEV_XID_ERROR`

  - Only for the following error codes: `46, 48, 54, 62, 64, 74, 79, 95, 109, 110, 119, 120, 136, 140, 142, 143, 151, 155, 156, 158, 159, 167, 171, 172`
- `DCGM_FI_DEV_ROW_REMAP_FAILURE`
- `DCGM_FI_DEV_ROW_REMAP_FAILED`
- `DCGM_FI_DEV_THRESHOLD_SRM`
- `DCGM_FI_DEV_SRAM_EXCEEDED`
- `DCGM_FI_DEV_GET_GPU_RECOVERY_ACTION`
- `DCGM_FI_DEV_GPU_RECOVERY_ACTION`

**AMD** (reported by the [AMD Device Metrics Exporter](https://github.com/ROCm/device-metrics-exporter)):

- `GPU_ECC_UNCORRECT_TOTAL`
- `GPU_ECC_UNCORRECT_SDMA`
- `GPU_ECC_UNCORRECT_GFX`
- `GPU_ECC_UNCORRECT_MMHUB`
- `GPU_ECC_UNCORRECT_ATHUB`
- `GPU_ECC_UNCORRECT_BIF`
- `GPU_ECC_UNCORRECT_HDP`
- `GPU_ECC_UNCORRECT_XGMI_WAFL`
- `GPU_ECC_UNCORRECT_DF`
- `GPU_ECC_UNCORRECT_SMN`
- `GPU_ECC_UNCORRECT_SEM`
- `GPU_ECC_UNCORRECT_MP0`
- `GPU_ECC_UNCORRECT_MP1`
- `GPU_ECC_UNCORRECT_FUSE`
- `GPU_ECC_UNCORRECT_UMC`
- `GPU_ECC_UNCORRECT_MCA`
- `GPU_ECC_UNCORRECT_VCN`
- `GPU_ECC_UNCORRECT_JPEG`
- `GPU_ECC_UNCORRECT_IH`
- `GPU_ECC_UNCORRECT_MPIO`

## Observe Node Readiness Conditions

To check the current node conditions and taints on a node, run the following command, replacing `<node-name>` with the name of the node:

```shell
kubectl describe node <node-name>
```

The output includes a `Conditions` section and a `Taints` section. A node that is pending readiness shows one or more `readiness.k8s.io/` taints with the corresponding condition set to `False`. For example:

```text
Taints:             readiness.k8s.io/DOKSAmdGpuReady=pending:NoSchedule
...
Conditions:
  Type                             Status
  ----                             ------
  doks.kubernetes.io/AmdGpuReady   False
  Ready                            True
```

The `kubectl` field selector does not support taint keys on nodes, so use a custom-columns output to list every node alongside its taint keys:

```shell
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'
```

To narrow the list to nodes carrying a specific readiness taint, filter the output with `grep`:

```shell
kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep readiness.k8s.io/DOKSAmdGpuReady
```

When all conditions required by a rule are `True`, the controller removes the corresponding `readiness.k8s.io/` taint and the node becomes schedulable.

## Override GPU Health Metrics

The reporter uses a fixed default list of failure metrics, but you can add or remove metrics without redeploying the DaemonSet. To customize the metrics, create a ConfigMap named `node-readiness-condition-gpu-metrics` in the `kube-system` namespace with a `metrics.yaml` key.

Create a file named `node-readiness-gpu-metrics.yaml` with the following contents:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-readiness-condition-gpu-metrics
  namespace: kube-system
data:
  metrics.yaml: |
    additionalMetrics:
      - MY_ADDITIONAL_METRIC
    disabledMetrics:
      - MY_DISABLED_METRIC
```

Apply the file to your cluster:

```shell
kubectl apply -f node-readiness-gpu-metrics.yaml
```

The reporter watches the ConfigMap and applies changes without restarting. Metrics listed under `additionalMetrics` are added to the checked set. Metrics listed under `disabledMetrics` are removed from the checked set. Both fields are optional. Deleting the ConfigMap resets the reporter to the default metric list.