For AI agents: The documentation index is at https://docs.digitalocean.com/llms.txt. Markdown versions of pages use the same URL with index.html.md in place of the HTML page (for example, append index.html.md to the directory path instead of opening the HTML document).

DigitalOcean Kubernetes (DOKS) runs a node readiness controller as a managed component that prevents pods from being scheduled onto nodes until all required components are healthy. The controller monitors custom node conditions and manages NoSchedule taints automatically, removing them once the associated conditions are met.

How the Node Readiness Controller Works

The controller is built on the upstream node-readiness-controller project. It introduces a NodeReadinessRule custom resource that declares:

• Conditions: One or more node conditions that must all have the required status.
• Taint: The NoSchedule taint to apply to the node while conditions are unmet.
• Node selector: Which nodes the rule targets, using label selectors.
• Enforcement mode: bootstrap-only evaluates the rule only during node startup; continuous re-evaluates it throughout the node’s lifetime.

When a node does not satisfy all conditions in a rule, the controller applies the rule’s taint to the node. When all conditions are satisfied, the controller removes the taint and the node becomes available for scheduling.

The controller uses only taints with keys prefixed by readiness.k8s.io/. These taints are distinct from the GPU device plugin taints (amd.com/gpu:NoSchedule, nvidia.com/gpu:NoSchedule) described in Automatic Application of Labels and Taints to Nodes.

Managed Node Readiness Rules

DOKS deploys three NodeReadinessRule resources, all using continuous mode so they re-evaluate conditions throughout the node lifecycle, not only during bootstrap. The controller and the NodeReadinessRule custom resource definition are present on all clusters, but the three managed rules require Kubernetes 1.33.9-do.6, 1.34.5-do.6, 1.35.1-do.6, 1.36.0-do.0, or later. Clusters on earlier patch releases run the controller without the managed rules until you upgrade.

DOKS sets the doks.kubernetes.io/CiliumAgentReady and doks.kubernetes.io/CPBridgeReady conditions from the health of the Cilium agent and the control plane bridge (cpbridge) running on each node. The GPU conditions are set by a separate reporter, as described in the next section.

GPU Node Health Conditions

DOKS runs a GPU node condition reporter as a DaemonSet on GPU worker nodes. It sets the doks.kubernetes.io/AmdGpuReady or doks.kubernetes.io/NvidiaGpuReady node condition by querying the metrics endpoint of the GPU metrics exporter running on the same node.

The reporter evaluates the condition as follows:

• If the metrics endpoint is reachable and returns a 2xx status code, and none of the tracked failure metrics have a value greater than zero, the condition is set to True.
• If the endpoint is unreachable, returns a non-2xx status code, or any tracked failure metric has a value greater than zero, the condition is set to False.

The reporter checks the following metrics:

NVIDIA (reported by the DCGM Exporter):

• DCGM_FI_DEV_XID_ERRORS
  • Only for the following error codes: 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 33, 34, 35, 36, 42, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 62, 64, 65, 73, 74, 79, 81, 87, 90, 91, 95, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 122, 123, 124, 125, 136, 138, 140, 142, 143, 151, 155, 156, 158, 159
• DCGM_FI_DEV_ECC_DBE_VOL_TOTAL
• DCGM_FI_DEV_ECC_DBE_AGG_TOTAL
• DCGM_FI_DEV_RETIRED_DBE
• DCGM_FI_DEV_ROW_REMAP_FAILURE

AMD (reported by the AMD Device Metrics Exporter):

• GPU_ECC_UNCORRECT_TOTAL
• GPU_ECC_UNCORRECT_SDMA
• GPU_ECC_UNCORRECT_GFX
• GPU_ECC_UNCORRECT_MMHUB
• GPU_ECC_UNCORRECT_ATHUB
• GPU_ECC_UNCORRECT_BIF
• GPU_ECC_UNCORRECT_HDP
• GPU_ECC_UNCORRECT_XGMI_WAFL
• GPU_ECC_UNCORRECT_DF
• GPU_ECC_UNCORRECT_SMN
• GPU_ECC_UNCORRECT_SEM
• GPU_ECC_UNCORRECT_MP0
• GPU_ECC_UNCORRECT_MP1
• GPU_ECC_UNCORRECT_FUSE
• GPU_ECC_UNCORRECT_UMC
• GPU_ECC_UNCORRECT_MCA
• GPU_ECC_UNCORRECT_VCN
• GPU_ECC_UNCORRECT_JPEG
• GPU_ECC_UNCORRECT_IH
• GPU_ECC_UNCORRECT_MPIO

Observe Node Readiness Conditions

To check the current node conditions and taints on a node, run the following command, replacing <node-name> with the name of the node:

kubectl describe node <node-name>

The output includes a Conditions section and a Taints section. A node that is pending readiness shows one or more readiness.k8s.io/ taints with the corresponding condition set to False. For example:

Taints:             readiness.k8s.io/DOKSAmdGpuReady=pending:NoSchedule
...
Conditions:
  Type                             Status
  ----                             ------
  doks.kubernetes.io/AmdGpuReady   False
  Ready                            True

The kubectl field selector does not support taint keys on nodes, so use a custom-columns output to list every node alongside its taint keys:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

To narrow the list to nodes carrying a specific readiness taint, filter the output with grep:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep readiness.k8s.io/DOKSAmdGpuReady

When all conditions required by a rule are True, the controller removes the corresponding readiness.k8s.io/ taint and the node becomes schedulable.

Override GPU Health Metrics

The reporter uses a fixed default list of failure metrics, but you can add or remove metrics without redeploying the DaemonSet. To customize the metrics, create a ConfigMap named node-readiness-condition-gpu-metrics in the kube-system namespace with a metrics.yaml key.

Create a file named node-readiness-gpu-metrics.yaml with the following contents:

apiVersion: v1
kind: ConfigMap
metadata:
  name: node-readiness-condition-gpu-metrics
  namespace: kube-system
data:
  metrics.yaml: |
    additionalMetrics:
      - MY_ADDITIONAL_METRIC
    disabledMetrics:
      - MY_DISABLED_METRIC

Apply the file to your cluster:

kubectl apply -f node-readiness-gpu-metrics.yaml

The reporter watches the ConfigMap and applies changes without restarting. Metrics listed under additionalMetrics are added to the checked set. Metrics listed under disabledMetrics are removed from the checked set. Both fields are optional. Deleting the ConfigMap resets the reporter to the default metric list.