Why does my load balancer show my Kubernetes node(s) as unhealthy?

If your nodes are showing as unhealthy, the Kubernetes service’s externaltrafficpolicy setting could be rejecting the load balancer’s health checks. This setting describes how nodes should respond to health checks from an external load balancer and can make nodes appear unhealthy if not set appropriately.

Most likely, your nodes are configured to reject health checks if they do not have a pod for that service running locally. If a node rejects a health check for a service, the load balancer shows the node as “Unhealthy” in the DigitalOcean Control Panel.

The service’s externaltrafficpolicy setting affects how nodes respond to these health checks when set with the following values:

  • Local - Any node not directly hosting a pod for that service will reject the request. To the load balancer, the request was rejected and therefore the node shows as “unhealthy”. Kubernetes considers these nodes healthy, but they will not field traffic for this service through the LoadBalancer.

  • Cluster - Nodes forward traffic to other nodes that are hosting pods for the service. Since the nodes are allowed to pass the traffic to other nodes, all of the node become valid endpoints as long as one pod is healthy.

To update this setting for a service, use the following command substituting <traffic-policy-value> for Local or Cluster:

kubectl patch svc <your-service> -p '{"spec":{"externalTrafficPolicy":"<traffic-policy-value>"}}'
Note
If using the Cluster setting, the original client IP address will be lost due to this extra network hop. To ensure fewer network hops, use the Local setting. For more information on externaltrafficpolicy, see the official Kubernetes documentation .