JVM Pods: OOMKills, CPU Throttling, and a Stuck Rolling Update
A Deployment runs 12 replicas of a Java API on a 6-node cluster. Each pod requests 100m CPU and 256Mi memory, with limits of 500m and 512Mi. The JVM is started with default heap settings. During peak traffic, pods are OOMKilled and CPU throttling is visible, while kubectl top shows nodes at about 40% CPU. An HPA targets 70% CPU utilization, but it rarely scales because the average across pods stays below target. A PodDisruptionBudget sets minAvailable: 11. A rolling update with maxSurge: 1 and maxUnavailable: 0 has been stuck for 20 minutes: new pods become Ready but old pods are not terminating. The team needs stable capacity during peak and predictable rollouts.
1. Explain the OOMKills, the CPU throttling despite low node utilization, and why the HPA rarely scales. Recommend resource and JVM changes.
Read the worked answer
The OOMKills come from the memory limit, not node pressure. The JVM with default heap settings sizes its heap from the container limit or from host memory depending on the JDK version and flags; if the heap plus metaspace, thread stacks, direct buffers, and GC overhead exceed 512Mi, the kernel OOM killer terminates the container. Set -XX:MaxRAMPercentage or explicit -Xmx to leave headroom, for example a heap around 70-75% of the container limit, and set -XX:MaxMetaspaceSize and -XX:MaxDirectMemorySize where relevant. Raise the memory limit only after measuring actual usage; a 256Mi request and 512Mi limit is likely too tight for a JVM service.
CPU throttling with nodes at 40% is expected when limits are enforced per pod: the pod gets at most 500m per 100ms period, so bursts are throttled even though the node has idle capacity. The 100m request also tells the scheduler the pod needs little CPU, so it may pack many pods onto a node. Raise the CPU request toward observed steady-state usage, for example 300-500m, and set the limit high enough to avoid severe throttling, or remove the CPU limit if the policy allows and rely on requests for scheduling.
The HPA computes utilization against requests, not limits. With a 100m request, actual usage may be well above 70m but still below the 70% threshold relative to the request in a way that does not reflect saturation; more importantly, throttling caps measured usage at the limit. Fix requests first, then configure the HPA with realistic targets and behavior such as scale-up stabilization windows. Consider scaling on a custom metric such as request concurrency or latency if CPU is a poor proxy.
2. Why is the rolling update stuck even though new pods become Ready, and how does the PodDisruptionBudget interact with it?
Read the worked answer
With maxUnavailable: 0, the Deployment will not terminate an old pod until a new pod is available and the total available count stays at 12. The PDB with minAvailable: 11 allows at most one pod to be voluntarily disrupted at a time. The rolling update is a voluntary disruption, so the eviction or deletion path can be constrained by the PDB. If the PDB's current healthy count is 11 because one pod is unavailable for another reason, no further old pod can be removed, and the rollout stalls. New pods becoming Ready does not automatically free an old pod if the controller's availability calculation or the eviction API sees the budget exhausted.
Check kubectl get pdb for ALLOWED DISRUPTIONS. If it is 0, find the unavailable pod: a crashloop, a failed readiness probe, or a pod on a draining node. Also check whether the rollout uses the Deployment controller's delete path or an eviction-based tool; the Deployment controller deletes pods directly, but cluster policies or autoscalers may interact. A maxUnavailable: 0 plus maxSurge: 1 rollout requires one extra pod of capacity; if the cluster cannot schedule the surge pod due to requests, the rollout waits even if pods look Ready.
For predictable rollouts, set maxUnavailable: 1 and maxSurge: 1 if the service can tolerate one fewer pod, and set the PDB to minAvailable: 11 consistently. Ensure the cluster has capacity for surge pods based on the corrected CPU and memory requests. Monitor kubectl rollout status and PDB allowed disruptions during the rollout.
3. The team wants stable peak capacity and predictable rollouts without overprovisioning the cluster. Propose concrete changes and the order in which you would apply them.
Read the worked answer
First fix the application and resource contract, because HPA and rollout behavior depend on accurate requests. Measure actual CPU and memory under peak, set memory requests close to steady-state usage and limits with JVM headroom, and set CPU requests near observed usage rather than 100m. Re-deploy and confirm OOMKills and throttling drop. Do not tune the HPA before requests are realistic.
Second, adjust the rollout strategy and PDB together. If the service can tolerate one unavailable pod, use maxUnavailable: 1 and maxSurge: 1; keep the PDB at minAvailable: 11. Verify that the cluster can schedule surge pods with the new requests. If capacity is tight, use a surge pod that fits or temporarily scale the node group.
Third, configure the HPA with a target based on corrected requests, and add behavior policies to avoid flapping. Consider a second metric such as latency or concurrency, since CPU may not track JVM saturation well. Set a minimum replica count that covers peak baseline, and use a cluster autoscaler with node pools that match the pod requests. Validate with a load test: confirm that the HPA scales before latency degrades, that no pod is OOMKilled, and that a rolling update completes while the PDB still allows one disruption.