Mixed-Version NKP Workload Cluster After a Blocked Drain
A platform team runs Nutanix Kubernetes Platform (NKP) 2.6 with one management cluster and four workload clusters on AHV. The production workload cluster has three control-plane nodes, six workers, Kubernetes v1.26.9, Cilium CNI, and the Nutanix CSI driver across 12 production namespaces. They schedule an upgrade to v1.27.7 in a four-hour change window. Preflight checks pass. During rollout, four workers upgrade, but two workers stay on v1.26.9. The machine deployment reports a drain failure against a monitoring namespace whose PodDisruptionBudget sets minAvailable: 2 with exactly two replicas. The window closes, leaving the cluster mixed-version. Application pods are healthy, but the team wants to complete the upgrade next weekend without reducing monitoring coverage or breaking the NKP management cluster supported version skew.
1. What does the mixed-version state imply for supportability and workload safety, and what should the team verify first before retrying the upgrade?
Read the worked answer
A temporary mixed kubelet version during a rolling upgrade is normal, but leaving it across a full week is not. The first checks are not application health; they are version skew and component compatibility.
Verify that the NKP management cluster release supports the target workload Kubernetes version, that all control-plane nodes report v1.27.7, and that the two remaining workers are the only v1.26.9 nodes. Check Cilium, CSI node plugins, and any admission webhooks for version-specific DaemonSets or container images. Confirm the MachineDeployment is paused rather than failed, and read the exact drain event.
The drain is blocked by the eviction API, not by the CNI. With minAvailable: 2 and two replicas, the PDB allows zero voluntary disruptions. That is a deliberate availability control, so deleting the PDB is not a safe fix. Confirm the PDB selector only matches the monitoring pods; a broad selector can block unrelated drains. Check whether the two old workers host monitoring replicas and whether anti-affinity prevents rescheduling. The cluster is supportable only within documented skew; complete the upgrade or roll back deliberately, do not leave it indefinitely.
2. The team proposes deleting the PodDisruptionBudget during the next window. Evaluate that proposal and give a safer operational plan.
Read the worked answer
Deleting the PDB removes the guardrail that prevents simultaneous eviction of both monitoring replicas. During a node drain, the scheduler may move one pod and then the second, leaving a gap in coverage exactly when the platform is changing. It also sets a precedent that availability controls are disposable.
A safer plan is to make the workload temporarily eviction-tolerant. Scale the monitoring deployment to three replicas, keep minAvailable: 2, and ensure anti-affinity spreads replicas across at least three workers. That allows one eviction at a time. If scaling is impossible, change the PDB to maxUnavailable: 1 for the window with explicit application-owner approval, not deletion. Verify that the PDB status shows disruptionsAllowed: 1 before draining.
Drain one old worker at a time using the eviction API, not force deletion. Watch the PDB, pod rescheduling, Cilium health, and CSI node plugin readiness between nodes. After each drain, confirm the replacement pod is Running and Ready on a v1.27.7 node. Keep the old node cordoned until the workload is stable. If the PDB still blocks, stop and reassess rather than bypassing it with force flags.
3. Sequence the remaining upgrade and define rollback criteria that the team can defend in a change review.
Read the worked answer
Sequence should be management cluster compatibility first, then workload control plane, then workers one at a time. Confirm the management cluster is healthy and at a release that supports the workload target. Do not start another workload cluster upgrade in parallel. For the remaining two workers, cordon, drain with eviction, confirm the node is empty, upgrade, uncordon, and wait for Cilium, CSI, and monitoring pods to become Ready before moving to the next node.
Rollback criteria should be written before the window. Stop and roll back if the control plane cannot maintain quorum, if the API server error rate exceeds an agreed threshold, if Cilium endpoints are not programmed on upgraded nodes, or if CSI volumes cannot attach to rescheduled pods. Kubernetes does not support downgrading a control plane in place, so the real rollback is restoring the etcd snapshot and rebuilding the affected nodes, or failing over to a standby cluster. The team must verify that an etcd snapshot and a tested restore procedure exist before starting. Document that mixed worker versions are acceptable only during the change, not as a steady state.