Auto Scaling That Scales Out but Never In
An API service runs in a scaling group with a minimum of 4 and maximum of 30 ECS instances. The scaling rule is target tracking on average CPU at 50%. After a marketing campaign the group sits at 28 instances with average CPU at 18%. It has not scaled in for three days. The team also notices that instances added by scaling take about 6 minutes before they pass SLB health checks, and during that time error rates rise.
1. List the likely reasons the group will not scale in, and how you would confirm each.
Read the worked answer
Candidates:
- Cooldown or scaling suspension. A long cooldown after the last scaling activity, or a suspended scale-in process, prevents removal. Confirm in the scaling group's scaling activity log and configuration.
- Health check grace and instance health. If instances are considered unhealthy or still initializing, they may not be eligible for removal. Check instance health status in the group.
- Protected instances. Instances manually set to protected, or attached outside the group, are never removed. Check the protected-instance list.
- Target tracking hysteresis. Target tracking scales in only when the metric is below target by a margin for a sustained period; a metric that fluctuates can keep it from acting. Review alarm history.
- Metric mismatch. If the rule tracks average CPU but a subset of instances is busy, the average may not reflect reality - but here 18% is well below 50%, so this is less likely the blocker.
- Maximum/minimum and balancing. The group is at 28, below max 30, so it is not pinned at maximum.
Confirm via the scaling activity log first: it usually names the reason a scale-in was skipped.
2. Explain why newly added instances take ~6 minutes to become healthy and how that interacts with scaling behavior.
Read the worked answer
The delay is boot and warm-up time: OS start, agent registration, application start, and possibly image pull or cache warm-up. Until the instance passes SLB health checks it receives no traffic, so it contributes zero capacity while the existing instances absorb the load - which is why error rates rise during scale-out.
This interacts with scaling in two ways:
- Scale-out is late. By the time new capacity is healthy, the spike may be over, leaving the group over-provisioned and slow to scale back in.
- Health check grace period must be long enough that instances are not marked unhealthy and replaced during warm-up, but a long grace period also delays detection of genuinely bad instances.
Mitigations: pre-bake the application into a custom image, use startup scripts or user data to reduce boot work, add a warm-up or connection-draining setting on the SLB, and consider scheduled scaling to add capacity before the known campaign peak rather than reacting to it.
3. What changes would you make to the scaling configuration to keep the fleet closer to actual demand without causing thrashing?
Read the worked answer
- Add scheduled scaling for known campaign windows so capacity is in place before demand arrives, rather than waiting for CPU to rise.
- Tune target tracking with a sensible scale-in cooldown and a margin so the group does not oscillate around the target. Scaling in slowly is usually safer than scaling in aggressively.
- Pick a signal tied to the bottleneck. If latency, not CPU, is the constraint, track a request-count or latency metric per instance.
- Reduce warm-up time through custom images and pre-initialized caches so reactive scaling is effective.
- Set a realistic minimum based on the true baseline; the current floor of 4 may be too low if baseline load requires more.
- Review the maximum against cost budget; 30 may be more headroom than needed if scheduled scaling covers known peaks.
Then observe for a full weekly cycle before further changes, and confirm scale-in actually occurs by checking the activity log.