HPA That Never Fires on a Latency-Bound Java Service
A logistics company runs an order-tracking Java service on CCE. Eight pods are spread across two AZs; each declares requests of 1 vCPU and 2 GiB with limits of 2 vCPU and 4 GiB. The HPA is configured for CPU at a 60% target, minimum 4 and maximum 12 replicas. The node pool has four 8-vCPU nodes and cluster autoscaler is disabled. During a marketing campaign, request volume doubles for 40 minutes. Latency p99 climbs from 400 ms to 3.1 s and the service stays at eight pods throughout; CPU utilisation on the pods peaks at 45%. Redis p99 is 8 ms and the RDS for MySQL instance peaks at 50% CPU with 300 connections against a 1,000 limit. Application metrics show the request thread pool at 200/200 active with a growing queue. The container image is 1.8 GB and is stored in an SWR repository in a different region; a cold pod takes 95 s to become ready.
1. Explain why the HPA did not scale, covering every blocking factor you can identify from the data.
Read the worked answer
There are three independent reasons, and fixing only one leaves the incident in place.
1. The metric never reaches its threshold. HPA computes utilisation as usage ÷ request. The pods request 1 vCPU but only use about 0.45 vCPU under peak load, so utilisation reads 45% against a 60% target. The pods are not CPU-starved - they are thread-starved, waiting on downstream or on queueing. A CPU-based HPA is structurally blind to a latency-bound bottleneck, so no amount of tuning the target fixes it. The inflated request (1 vCPU for a workload using 0.45) actively hides the problem by depressing the percentage.
2. There is nowhere to schedule. Cluster autoscaler is disabled and the node pool is four 8-vCPU nodes. Scaling from 8 to 12 pods needs roughly 4 additional vCPU of requests plus system overhead; without autoscaling, any pod the HPA did create would sit Pending.
3. Scale-out would be too slow anyway. A 95 s cold start (1.8 GB image pulled cross-region) against a 40-minute campaign means a new pod spends the first minutes of the spike not serving traffic. Even a perfect scaling signal would arrive too late.
The deeper point: the symptom is throughput, the cause is concurrency and downstream dependency behaviour, and the configured autoscaler measures the wrong resource for this workload.
2. Redesign the scaling and capacity configuration. Be specific about signals, thresholds and supporting changes.
Read the worked answer
Signal. Scale on the thing that is actually saturated - requests in flight, queue depth, or QPS per pod - exposed to CCE via AOM/Prometheus and used as a custom-metric HPA. A workable rule: target a queue depth of ~10 and QPS per pod derived from a load test at p99 ≤ 500 ms. Keep CPU as a secondary guardrail only, with a low target (~50%) because the real per-pod usage is ~0.45 vCPU.
Requests and limits. Set requests to observed p95 usage, e.g. 500m CPU and 2 GiB, and keep limits above that so the JVM has headroom. Right-sizing the request makes CPU utilisation readings meaningful and improves bin-packing.
Capacity. Enable cluster autoscaler on the node pool with a maximum of 8 nodes, and set the HPA max above the immediate need (e.g. 16) so the autoscaler, not the HPA ceiling, is the binding constraint. Add anti-affinity across AZs and a PodDisruptionBudget so scaling and node drains do not concentrate pods.
Startup time. Mirror the image to SWR in the same region as the cluster, add image pre-pull or a warm standby pool, and tune readiness probes so a pod is marked ready only when the thread pool and downstream connections are initialised. Target cold start under 30 s.
Downstream protection. Size the JDBC pool and Redis connections per pod so that 16 pods do not exceed the RDS connection limit; raise the RDS max_connections with the scaling ceiling, not after it.
3. How would you validate the redesign before the next campaign, and what would make you reject the design?
Read the worked answer
Validation plan.
- Run a stepped load test to 2× the campaign peak on a staging cluster with identical node types, image and downstream latency injection (add 8 ms to Redis, 20 ms to RDS).
- Record time to first additional ready pod and time to steady state; both must complete inside the 40-minute demand window, ideally under 3 minutes.
- Confirm p99 stays under the SLO through the scale-out ramp, not just at steady state - that is where thread pools are still filling.
- Verify the autoscaler adds nodes before pods go Pending, and that scale-in does not drop below the minimum during the decay.
- Repeat with one AZ's pods removed to confirm cross-AZ resilience under load.
- Compare cost at 12 vs 16 pods to confirm the headroom is affordable.
Rejection criteria. Reject the design if: any of the trigger signals cannot be produced without a custom exporter that itself becomes a single point of failure; if time to ready exceeds ~2 minutes even with the local image; if the custom-metric HPA oscillates (scale up/down flapping) because the metric is noisy and no stabilisation window is set; or if scaling the pods pushes RDS connections past 80% of the limit, in which case the correct answer is a connection proxy or read replicas, not more pods. Autoscaling that moves the bottleneck downstream has not solved anything.