Cost Spike After an Autoscaler Change
An online ticketing platform runs on a managed instance group with autoscaling. Last month's bill jumped 40%. The team recently changed the autoscaler to scale on CPU utilization with a target of 50% and added a new HTTP(S) load balancer health check. Symptoms: instances stay at high count overnight when traffic is low, and average CPU sits near 20% during the day. The app has a warm-up period of about 90 seconds. There are no budget alerts configured. The team wants to know why instances are not scaling down and how to prevent recurrence.
1. Diagnose the most likely cause of the failure to scale down and propose fixes.
Read the worked answer
The symptom pattern (high instance count at low traffic, low average CPU) points to the autoscaler never observing a sustained low-utilization signal, most likely because:
- Health check / warm-up mismatch: the new health check may mark instances unhealthy or the 90-second warm-up causes the autoscaler to keep replacements running, inflating count. Set an appropriate initial delay so new instances are not counted as unhealthy during warm-up.
- Scale-in stabilization window: the autoscaler may be configured with a long stabilization window or conservative scale-down settings, so it holds capacity. Review and shorten it.
- CPU target too low (50%): with average CPU near 20%, the autoscaler should scale in, so if it is not, the signal is being overridden by the above settings or by per-instance utilization being masked.
Fixes: set warm-up/initial delay to ~90s, tune the stabilization window, and validate with autoscaler logs. Confirm the health check is not flapping.
2. What operational controls should be added to catch this earlier next time?
Read the worked answer
Add layered cost and health controls:
- Budget alerts at 50/80/100% of expected spend, routed to the on-call channel.
- Cloud Monitoring alert on instance count deviating from an expected band, and on autoscaler events.
- Dashboard showing instance count vs. request rate vs. CPU, so a divergence (count high, traffic low) is visible.
- Change review: require autoscaler and health-check changes to be tested in staging with a soak period before production.
These controls turn a silent 40% overspend into an early, attributable signal.