Autoscaling That Scales the Wrong Thing
A web application runs behind a load balancer with an autoscaling group of application servers. Scaling policy is based on average CPU across the group, targeting 60%. Traffic is bursty and arrives in short spikes of 30-90 seconds. During spikes, users report errors and slow pages even though the group eventually scales out. Investigation shows: (1) new instances take about 4 minutes to become healthy and receive traffic; (2) the load balancer's health check has a 30-second interval and 3-failure threshold; (3) average CPU across a large group barely moves when a spike hits only the newest instances; (4) the database connection pool is fixed at 100 connections and is already near saturation at current instance count. The team's monthly bill has risen 35% from over-scaling during quiet periods.
1. Explain why CPU-based scaling fails to protect users during these spikes, and what metric or mechanism would respond faster.
Read the worked answer
Two separate reasons the current policy fails:
- Lag between signal and capacity. Scale-out takes ~4 minutes to become healthy, but spikes last 30-90 seconds. Even a perfect signal cannot add capacity in time for this spike. Scaling protects against sustained load, not sub-minute bursts.
- Averaging hides the spike. Average CPU across a large group dilutes the load of the few instances actually serving the burst. The metric the policy watches is not the metric users experience.
Faster-responding options:
- Queue/latency-based signals (request queue depth, p95 latency, in-flight requests per target) react to user pain directly rather than to a lagging proxy.
- Pre-scaling / scheduled capacity for known burst windows, plus a floor of warm capacity sized to absorb the largest expected 90-second spike.
- Target-tracking on a leading indicator with a lower threshold and shorter evaluation window, accepting more churn.
The honest answer: autoscaling alone cannot fix a 90-second spike with a 4-minute boot. The design needs headroom or faster-starting capacity.
2. The bill rose 35% from over-scaling while users still saw errors. Diagnose this apparent contradiction and recommend fixes.
Read the worked answer
This is not a contradiction - it is the signature of oscillating, mis-targeted scaling.
- A 30-second health check with a 3-failure threshold means an instance can take ~90+ seconds just to be declared unhealthy, and slow-to-start instances may be counted as unhealthy and replaced, churning capacity.
- Averaged CPU undershoots during real spikes, so the group over-provisions during quiet periods chasing a target it can't hit cleanly, then thrashes.
- The database pool at 100 connections is the real ceiling. Adding app instances beyond what the pool supports produces connection errors - more instances, more errors, more cost, no better user experience.
Fixes:
- Bound scale-out by the database. Treat the connection pool as the capacity limit; add a connection proxy/pooler or raise the ceiling deliberately before scaling app tier further.
- Tune health checks to match boot time (appropriate grace period) so instances aren't wrongly killed.
- Use a better scaling metric and set a sane minimum/maximum so quiet-period over-provisioning stops.
- Add a warm pool or scheduled floor for known bursts instead of reactive scaling.
Net: the cost and the errors share a root cause - scaling the wrong resource against the wrong signal.