I85PREP
⌘ K
KLCP study materials
KLCP study resources

HPA Oscillation Under Bursty Load

KLCP · Put your knowledge to work in context.

All case studies

HPA Oscillation Under Bursty Load

An API Deployment runs 4-20 replicas behind an HPA targeting 60% CPU utilization. Each pod requests 500m CPU. During a daily batch job, CPU spikes for ~90 seconds, then drops. The HPA scales out to 18 pods, then 5 minutes later scales back to 4, repeating every cycle. Pod startup takes ~25s (image pull + JVM warmup). Users report 502s during the scale-up window because new pods aren't Ready when traffic arrives. The team also notices the HPA sometimes scales down even while CPU is still elevated. Metrics-server scrapes every 15s; the HPA has default --horizontal-pod-autoscaler-downscale-stabilization of 5m.

1. Explain the two distinct problems: the 502s during scale-up and the scale-down behavior during elevated CPU.

Read the worked answer

502s during scale-up: The HPA reacts to CPU within its sync period (~15s), but new pods take ~25s to become Ready. During that gap, the Service endpoints still point at the old (saturated) pods, which queue requests and time out. The HPA adds capacity that can't serve traffic yet, so the burst still causes errors. The fix is not more aggressive HPA scaling but readiness-aware capacity: ensure the Service only routes to Ready pods (it does by default) and add a PodDisruptionBudget plus a pre-scaling signal (e.g., scale on a custom metric like RPS or queue depth that leads CPU, or add a small buffer via behavior.scaleUp policies).

Scale-down during elevated CPU: The HPA computes a recommendation from the current metric average, but the downscale stabilization window (5m) keeps the highest recent recommendation. If CPU briefly dipped (e.g., between metric scrapes, or because the batch job paused), the recommendation drops and the stabilization window may still allow a scale-down if the lower value persists. Also, the HPA uses the average across all pods; if new pods are idle (not yet receiving traffic), they drag the average down, making the HPA think it's over-provisioned. That's the real culprit: unready/idle pods dilute the utilization average.

2. Recommend concrete HPA behavior settings and a complementary change to stop the oscillation.

Read the worked answer

  1. Tune behavior: Set scaleDown.stabilizationWindowSeconds: 600 (10m) so short dips don't trigger scale-down, and add a scaleDown policy with selectPolicy: Min and a periodSeconds: 60 to limit how fast replicas drop. For scale-up, add a policy allowing a burst (e.g., pods: 4 per 30s) so the first response to the batch job is fast.

  2. Fix the metric dilution: The HPA averages CPU over all pods including unready ones. Use a metric that reflects actual load (RPS per pod) or ensure new pods warm up before counting. A startupProbe won't help here; the issue is the averaging, so prefer request-based autoscaling (e.g., a custom metric) for bursty traffic.

  3. Reduce warmup time: Pre-pull the image (DaemonSet or image cache) and use a smaller base image to cut the 25s to <10s, shrinking the unready window.

  4. Add a buffer: Set minReplicas to 6 so the baseline absorbs the first ~30s of the burst without waiting for scale-up. Combined with a longer downscale window, the cluster stops flapping between 4 and 18.