CCOA (IJCAHPO): The EKS Upgrade That Stalled at 60 Percent

The PDB arithmetic. Allowed disruptions equals ready replicas minus minAvailable. With 2 replicas and minAvailable 2, that is 0, permanently. The eviction API refuses every eviction, kubectl drain retries until its timeout, and the node is stuck. Drain is behaving correctly; the PDB is a self-inflicted deadlock.

Fixes for workload PDBs:

• Use maxUnavailable: 1 instead of minAvailable: 2 for small deployments, so one pod can always move.
• Where minAvailable is genuinely needed (for example 90% availability on a 10-replica service), ensure replicas exceed minAvailable by at least 2 and that the deployment has surge capacity on other nodes.
• Ensure the deployment itself is not pinned by anti-affinity to the node being drained.
• Add a pre-flight check in the upgrade job: for each PDB, compute allowed disruptions and fail the batch before draining if any workload has 0 and cannot be scaled up.

CoreDNS specifically. Two replicas with no topology constraints can co-locate. Use at least three replicas (or two per zone on large clusters), topologySpreadConstraints on topology.kubernetes.io/zone and kubernetes.io/hostname with whenUnsatisfiable: ScheduleAnyway so a drain cannot cluster them, a PDB of maxUnavailable: 1, and priorityClassName: system-cluster-critical. Give CoreDNS explicit CPU requests so the autoscaler can react, and set --max-surge/--max-unavailable on the CoreDNS deployment rather than naked replica counts.

Operational rule: drain with --timeout=300s --delete-emptydir-data --ignore-daemonsets, alert if a node stays cordoned more than 10 minutes, and treat a blocked drain as a stop-the-line event rather than something to wait out. Three hours of a stalled upgrade is far more dangerous than pausing and fixing the PDB.

Why CPU is the wrong signal. The HPA scales on the average utilization of CPU requests, which is a proxy for load, not a measure of user impact. At 55% average CPU the service already breaches p99 because the bottleneck is connection-pool saturation: requests queue for a database connection while CPUs sit partly idle. The HPA will not add a pod until 70%, which is past the SLO. Worse, if requests are missing or wildly over-provisioned, the utilization percentage is meaningless: a pod with a 2-core request that actually uses 0.6 cores reads as 30% while it is fully saturated on I/O. CPU throttling from a low limit can also inflate latency at low average utilization.

Use a load signal closer to the bottleneck. Scale on in-flight requests (concurrency) per pod, or requests per second per pod, using prometheus-adapter or KEDA. Target concurrency such that p99 stays under 400 ms with headroom, for example 60% of measured saturation concurrency. If the pool is the true constraint, scale on pool wait time or queue depth. Keep CPU as a secondary metric only if it correlates.

Configuration that stops the oscillation. The 4-to-12-to-4 cycle within 10 minutes is classic flapping: scale-up happens, load redistributes, average utilization drops, scale-down fires, load returns. Set behavior.scaleDown.stabilizationWindowSeconds: 300 or higher, and add a scale-up policy limiting to 2 pods or 100% per 60 s so a spike does not triple capacity. Keep scale-up stabilization at 0 for responsiveness.

Fix the real bottleneck too. Right-size connection pools per pod against database max_connections, add timeouts and circuit breakers, and set CPU requests from observed p95 usage so the autoscaler has a truthful denominator. A scaling policy cannot fix a pool that saturates below the SLO.

Before touching nodes. Confirm the control plane is already at the target minor version, and only ever move one minor at a time (1.26 to 1.27, then 1.27 to 1.28). Run an API deprecation scan (kubent, pluto) against every manifest and Helm chart; anything still on removed APIs will fail to schedule. Verify the VPC CNI, CoreDNS, kube-proxy, autoscaler, and CNI/CSI driver versions support 1.28.

Fix the blockers found in this incident. Correct the PDBs, spread CoreDNS, add CPU requests to the workloads that lack them, and confirm the autoscaler is healthy and can scale each node group. Increase Spot diversity (instance families and AZs) or switch to capacity-optimized allocation to reduce reclaims; 7 lost nodes mid-upgrade is a signal that the node group is too concentrated.

Blue/green node groups. Create a new node group at the target version with the same taints and labels, cordon and drain the old one in batches of 10-15% of nodes, never more than one AZ at a time. Let the autoscaler manage capacity during the drain so scheduling pressure is absorbed by the new group. Keep batch drain under 10 minutes; if a batch exceeds it, stop and inspect rather than force-delete pods.

Halt criteria, written down in advance: any drain blocked over 10 minutes, CoreDNS not fully ready, error-budget burn above the threshold for 15 minutes, any workload unable to schedule within 5 minutes, or more than one node of a stateful set unavailable.

Rollback. The control plane cannot be downgraded, so rollback is at the node layer: keep the old node group cordoned but intact until the new one has served a full business peak, then delete it. If the new nodes misbehave, uncordon the old group, drain the new one, and reschedule. That only works if the API deprecation scan was done first; otherwise the old nodes cannot run the workloads the new control plane accepts.