TL;DR

• What changed: Kubelet v1.31 now filters Services with spec.clusterIP to skip headless Services.
• Why it matters: If a node is upgraded to 1.31 while the API server is still 1.30, that older API server rejects the new field‑selector and pods on the node remain Pending.
• Quick fix: Upgrade all control‑plane nodes to v1.31 first (verify with kubectl version --short | grep Server) or roll back the Kubelet if you already upgraded out‑of‑order.

NOTE: You must upgrade the Kubernetes control plane to v1.31 and confirm the API server reports v1.31 before upgrading any worker nodes or Kubelets.
IMPACT: If a Kubelet is upgraded to v1.31 while its API server is still v1.30, pods on that node will fail to start because the older API server cannot parse the new spec.clusterIP selector. Upgrading in the wrong order expands the blast radius to the entire cluster and can cause downtime for all workloads.

What Changed & Why It Matters

Kubernetes 1.31 introduced a new field selector for Services on the clusterIP and type fields. The Kubelet uses this selector (spec.clusterIP!=None) to avoid watching headless Services and reduce memory usage. 

This change means a Kubelet v1.31 will query the API server with that filter. If the API server is still on 1.30 or older (which doesn’t support filtering by spec.clusterIP), it will reject the request. In effect, the Kubelet can’t retrieve Service data, which disrupts its ability to start pods on that node (since it cannot set up environment variables and networking info from Services).

Impact Matrix

Detection Checklist

• Run kubectl get pods -A --field-selector status.phase=Pending to identify any pods stuck in Pending (especially on newly upgraded nodes).
• Run kubectl version --short | grep Server to confirm the API server is exactly v1.31.x before upgrading any nodes.
• Inspect Kubelet logs for fieldSelector "spec.clusterIP!=None" not supported.
• Look for a surge in API server 400 error codes (e.g. via metrics or logs) corresponding to Service list requests from Kubelets.

Deep Dive – Root Cause

E0518 10:32:44 … fieldSelector "spec.clusterIP!=None" not supported

The above error appears in the Kubelet logs when it fails to list Services. A Kubelet v1.31 issues a list/watch for Services with spec.clusterIP!=None. An older API server (v1.30) doesn’t recognize spec.clusterIP as a valid field selector and returns a 400 Bad Request error. This mismatch causes the Kubelet’s service watcher to crash out, preventing pods on that node from starting because required Service environment variables and cluster IP assignments can’t be initialized.

Safe Remediation & Upgrade Paths

1. Stop upgrading additional nodes and address the version skew immediately if you’ve encountered this issue.
2. Preferred – Upgrade every control‑plane node to v1.31, then resume worker‑node/Kubelet upgrades.
3. Alternative – Temporarily downgrade the Kubelet back to v1.30 on the affected node to restore compatibility. 
4. Mitigation – If you must stagger control‑plane nodes, pre‑enable the ControlPlaneKubeletLocalMode feature gate so each control‑plane Kubelet talks only to its own (already‑updated) API server.
5. Follow the Kubernetes version‑skew policy – control plane ≥ node versions at all times.

How Chkk Upgrade Copilot Helps

Chkk’s Upgrade Copilot not only analyzes release notes and changelogs but also identifies critical Upgrade Advisories based on real-world experiences from users and insights from our pre-verification process. This helps you proactively address potential risks or breakages during your Kubernetes upgrades or open-source project updates.

In addition to Upgrade Advisories, Chkk covers comprehensive tasks including Helm diffs, CRD diffs, dependency mapping, configuration validations, and deprecated API detection, providing thorough coverage of upgrade-related risks. 

Click the button below to book a demo and learn more.