GPU autoscaling fails to absorb twice-daily traffic spikes

Why CPU fails. The workload is GPU-bound. Each replica holds one request on the GPU for most of its service time while the CPU thread blocks on a CUDA call - that is waiting, not working, so CPU utilization stays low no matter how saturated the GPU is. CPU at 35% during a spike is not headroom; it is the signature of a blocker. The HPA is measuring the wrong resource, and its 60% target means it will only act when the CPU is doing more work than the GPU, which for this workload essentially never happens.

Replace with a demand metric, not a resource metric. The most direct signal is load per replica: either requests-in-flight (concurrency) or ingress queue depth. Concurrency is better because it is available instantly from the server itself and does not lag.

• KEDA with a Prometheus scaler on in-flight requests per replica, target around 55-60% of the measured sustainable concurrency. Derive that number from the 42 rps at 280 ms: roughly 12 requests concurrently in flight per replica at saturation, so target about 7.
• Keep a secondary scaler on DCGM GPU utilization at 70% as a backstop for workloads where concurrency is not observable.
• Set stabilizationWindowSeconds asymmetric: scale up with a 0-second window (react immediately, never damp a spike) and scale down with a 600-second window, so you do not thrash the GPU pool between the two daily peaks.

Sanity check: 1,200 rps at 42 rps per replica needs about 29 replicas. That is the number the HPA should have been driving toward and the number the node pool must be able to supply.

Arithmetic of the delay. Scaling from 6 to 29 replicas requires new nodes: about 3.5 minutes to provision plus about 4 minutes of cold start - 7.5 minutes before the first new replica serves traffic. The spike lasts 25 minutes and reaches full height within a minute or two of 09:00. Correct metrics get you to the right target, but the capacity arrives far too late to help the leading edge.

Layer 1 - predict instead of react. The spikes are at fixed times; this is a scheduled event, not a surprise. Use a KEDA cron scaler (or a scheduled HPA) to pre-scale to ~26 replicas at 08:40 and again at 12:40, then let the concurrency scaler handle the residual. Scheduled capacity is the only mechanism that beats a cold start.

Layer 2 - cut the cold start. Four minutes is dominated by image pull and model load with no cache. Pre-pull the image onto nodes with a DaemonSet; put the model on a node-local cache or a fast shared volume; consider an init container that warms it. Quantizing the model shrinks load time and GPU memory at once. Realistic target: under 60 seconds.

Layer 3 - keep warm headroom. Overprovision with low-priority pause pods that get preempted by real replicas, so a node is always ready and a new replica schedules in seconds rather than minutes.

Layer 4 - decouple the batch workload. More below, but it also reduces the peak the interactive path must absorb, which reduces how far you have to scale.

The core mistake is that latency-tolerant work shares a path with latency-critical work. Once bulk and interactive requests are in the same queue, the scheduler has no basis for prioritizing, and interactive p99 is determined by the bulk backlog.

Isolation. Split into two deployments: an interactive service and a batch scorer. Put them in separate node pools, or at minimum use Kubernetes priority classes so batch pods are evicted first under pressure. Run the batch deployment on spot instances - it is retryable and latency-tolerant, so interruption is cheap and the cost is 60-70% lower. The shared model can live in both; the code and the model artifact are the same, only the deployment and SLO differ.

Queueing. Put bulk uploads behind a durable queue (SQS, RabbitMQ, or Kafka) with a separate worker deployment that scales on queue depth and age, not on request rate. Workers can batch aggressively - batch size 32 instead of 1 - which is several times more GPU-efficient and is exactly the optimization you cannot make on the interactive path. This turns a 25-minute spike into a backlog the workers drain over 40 minutes, which nobody notices because results are emailed.

Admission control. Move the bulk traffic off the synchronous endpoint: the upload returns 202 Accepted with a job ID as soon as the document is persisted. Then add a per-tenant rate limit on the interactive endpoint, and a bounded queue with a maximum wait - beyond it, return 429 with Retry-After rather than letting a request sit past the SLO. Failing fast at 300 ms is a better experience than succeeding at 1.8 s, and it protects the majority of requests from the minority that are overloading you.

Result: the interactive path sees only interactive traffic, its scaling problem shrinks to roughly steady state, and the twice-daily spikes stop being an SLO event.