Autoscaling That Oscillates Under Load
A web application runs on an SLB in front of an ECS scaling group. The scaling group uses a target-tracking rule on average CPU at 60%, with a 60-second cooldown and a minimum of 2 and maximum of 30 instances. During a daily traffic peak, the team observes: CPU climbs, instances scale out, CPU drops sharply, instances scale in, CPU climbs again - repeating every few minutes. Users see elevated latency during the oscillations. The application has a 3-minute warm-up (JVM start plus cache fill) before a new instance serves traffic well. The team wants to stop the flapping without over-provisioning permanently.
1. Explain why the scaling group oscillates and why the 60-second cooldown is implicated.
Read the worked answer
The loop is a feedback oscillation caused by a mismatch between the control interval and the system's response time. When CPU rises, the group adds instances. Because new instances take 3 minutes to warm up, they do not contribute capacity immediately; but the scaling action (and the eventual warm instances) overshoots demand, so CPU falls. The 60-second cooldown then permits a scale-in before the system stabilizes, removing capacity that was actually needed, which drives CPU back up. The cooldown is shorter than the warm-up, so the controller reacts to transient signals instead of steady state. This is an under-damped control loop: gain too high, damping too low.
2. Recommend concrete configuration changes to stabilize scaling.
Read the worked answer
- Set the cooldown longer than warm-up - at least 3–5 minutes, so the controller observes the effect of a scaling action before deciding again.
- Use step or scheduled scaling for the known daily peak. A scheduled scale-out shortly before the peak avoids reacting to the ramp at all.
- Add a warm-up / lifecycle hook so new instances are not counted as healthy until they pass a readiness check. This aligns the controller's view with reality.
- Pre-warm capacity: keep a small buffer above minimum, or use a pre-baked image so JVM/cache warm-up is shorter.
- Add a scale-in stabilization window (or a longer scale-in cooldown than scale-out) so the group is slow to remove capacity and fast to add it.
- Consider a better metric than average CPU (e.g., request count per instance or queue depth) if CPU is noisy.
3. How would you validate the fix without waiting for the next daily peak?
Read the worked answer
Run a controlled load test that reproduces the peak ramp against a staging scaling group with the new settings, and watch instance count and latency over time. Confirm that scale-out happens before user latency degrades, that no scale-in occurs within the warm-up window, and that instance count settles rather than oscillating. Also review scaling activity logs from the previous peak to confirm the oscillation period matches the cooldown-plus-warm-up hypothesis. Only promote to production after the staging run shows a stable curve and acceptable cost (instance count vs. peak).