Autoscaling That Scaled Into a Wall
A containerized API runs on a managed Kubernetes cluster behind a load balancer. The service scales on CPU at 70% utilization, minimum 3 pods, maximum 30. During a marketing push, traffic triples in 90 seconds. CPU rises, pods scale out, but response latency climbs from 120 ms to 4 seconds and stays there for 20 minutes. Investigation shows: the database connection pool is fixed at 20 connections per pod, the database allows 200 total, and pods reached 30. New pods took 75 seconds to become ready because they download a 1.2 GB model at startup. The load balancer health check has a 5-second timeout. The team's first instinct is to raise the maximum pod count to 60.
1. Explain why raising the maximum pod count to 60 would likely make things worse, and identify the binding constraints.
Read the worked answer
More pods would multiply the pressure on the true bottleneck. At 30 pods with 20 connections each, the service already requests 600 connections against a database limit of 200, so most pods are contending for connections and queueing. Raising the ceiling to 60 would request 1,200, deepening the queue and lengthening latency further, while also multiplying the 1.2 GB startup download and the load on the database.
Binding constraints, in order of impact:
- Database connection ceiling (200). This caps useful concurrency regardless of pod count. The service is effectively limited to about 10 pods at 20 connections each unless connections are pooled or the limit is raised deliberately.
- Slow startup (75 s). Scaling out cannot help a 90-second traffic spike if new capacity is not ready for over a minute. The autoscaler reacts after CPU rises, then waits 75 s, by which time the spike has already saturated existing pods.
- Health check timeout (5 s) versus 75 s readiness. If readiness is conflated with liveness or the check is too aggressive, pods may be killed or marked unhealthy during startup, churning capacity.
- CPU-only scaling signal. CPU lags the actual symptom (latency), so the trigger fires late.
Raising the max is a non-fix because it adds consumers to a saturated resource. The right moves are connection pooling/proxying, pre-baking or caching the model to cut startup, and scaling on a leading signal.
2. Propose a remediation plan that addresses the 90-second spike without over-provisioning permanently.
Read the worked answer
The plan should combine faster readiness, a leading scaling signal, and protection of the database.
1. Cut startup time. Bake the 1.2 GB model into the image or mount it from a warm cache so pods start in seconds rather than 75. This is the highest-leverage change: it converts scale-out from useless to useful during a 90-second spike.
2. Protect the database. Put a connection pooler (e.g., a proxy that multiplexes connections) in front of the database so pod count no longer maps linearly to connections. Set a hard budget: total connections must stay under 200 with headroom. Configure per-pod pools small (e.g., 5) and let the pooler queue.
3. Scale on a leading signal. Use requests-per-second or in-flight requests or latency rather than CPU alone, and set a lower, earlier threshold so scale-out begins before saturation. Add a scheduled minimum bump during known marketing windows.
4. Fix health checks. Readiness should reflect ability to serve, with a startup probe that tolerates initialization; liveness should not kill pods that are merely warming. Align timeouts with realistic startup.
5. Pre-scale for known events. The marketing push was known; a scheduled scale-up 5 minutes before is cheaper and more reliable than reactive autoscaling.
6. Keep a sane max. Cap at the level the database and pooler can support (roughly 10–15 pods unless the pooler changes the math), and load-test to find it rather than guessing 60.