Flash Sale on a Fixed Auto Scaling Group: Why the Policy Never Fires in Time
A retail company serves its product-listing tier from eight ECS instances of flavor c6.large.2 (2 vCPU, 4 GB) in one Auto Scaling group behind a dedicated ELB. Normal traffic keeps average CPU near 40% and memory near 55%. Every Monday at 10:00 a 15-minute flash sale drives CPU to 95%, latency from 200 ms to 3.2 s, and 5% of requests returning HTTP 5xx. The scaling group is configured with min 8, max 12, and an alarm policy that adds one instance when CPU exceeds 70% for three consecutive 5-minute periods, followed by a 300-second cooldown. A new ECS needs 4 minutes to boot, and the Java application needs another 3 minutes to warm its cache before serving traffic. Finance has approved extra capacity for at most 14 instance-hours per week. One engineer proposes replacing the eight small instances with four c6.2xlarge.4 (8 vCPU, 16 GB) instances and raising max to 6.
1. Explain precisely why the current scaling policy cannot protect the Monday sale, and state what you would change.
Read the worked answer
Timeline arithmetic first.
- The alarm needs CPU above 70% for three consecutive 5-minute periods, i.e. 15 minutes of sustained load before any action is emitted. The sale itself lasts 15 minutes, so the trigger fires at best as the peak ends.
- Then 4 min boot + 3 min cache warm-up = 7 more minutes before the instance serves traffic.
- The 300-second cooldown then blocks any further action for 5 minutes after the first one.
- Earliest useful capacity: roughly 22 minutes after 10:00 - the sale is already over.
Capacity arithmetic. A step of +1 raises capacity from 8 to 9 instances, about 12.5%, while the load jumps from 40% to 95% CPU (roughly 2.4x). One instance cannot absorb that.
Fixes.
- Replace the reactive policy with a scheduled policy at 09:45 Monday and a periodic policy covering the weekly window, adding 4 instances ahead of the demand.
- Keep an alarm policy as a backstop but with a 1-minute period, 60% threshold and step +2.
- Cut cooldown to 120 s.
- Bake a pre-warmed private image (IMS) so cache warm-up happens at boot, and set the ELB health-check grace period above the boot time.
- Add scale-in stabilization so instances are not removed while the queue drains.
2. Evaluate the proposal to replace eight c6.large.2 instances with four c6.2xlarge.4 instances.
Read the worked answer
It solves nothing and makes scaling worse.
- Memory is not the bottleneck: 55% of 4 GB is about 2.2 GB used. The workload is CPU-bound, so a general-computing c6 flavor is already the right family; moving to 16 GB pays for memory the application never uses.
- Total vCPU rises from 16 to 32, but the load is not CPU-starved at the total level - it is starved per instance during the burst. Buying 2x capacity permanently to cover a 15-minute weekly spike is the expensive way to solve it; the AS group exists precisely to rent capacity by the hour.
- Scaling granularity collapses. With 4 instances, one step of +1 is 25% of the group and the maximum of 6 caps you at 48 vCPU; with 8 small instances a step of +2 is a smooth 25% and failure of one instance costs 12.5%, not 25%.
- Blast radius and boot cost: four large instances means losing one removes a quarter of capacity, and 8 vCPU instances take longer to boot and warm.
Recommendation: keep c6.large.2, or move to c6.xlarge.2 (4 vCPU, 8 GB) as a middle ground if per-instance agent overhead is high. Verify single-thread performance is not the real limit before changing flavor at all.
3. Before scaling to 12 instances, what must be verified, and how should the instances be billed?
Read the worked answer
Quota and dependency checks (all region-scoped):
- vCPU quota for the ECS flavor family in the region - a default quota can silently block the 9th instance.
- Free IP addresses in the subnet; ENI, EVS disk and snapshot quotas.
- ELB backend server limit and whether the listener's health check tolerates 7-minute warm-up.
- Downstream limits: database max_connections, NAT gateway SNAT connections, EIP bandwidth. Scaling the web tier onto a saturated database converts a latency problem into an outage.
Billing mix:
- The baseline 8 instances run 168 h/week and should be yearly/monthly billed.
- The burst instances exist ~15–30 h/week and should be pay-per-use, which also matches the 14 instance-hour budget: 4 instances x 1 h x 1 sale is 4 instance-hours, comfortably inside the cap only if the group scales back in promptly - which is why scale-in stabilization must be tuned, not disabled.
Also verify the metric itself. Confirm with the ELB that the bottleneck is application CPU and not TLS handshakes or slow database queries; if p95 latency degrades before CPU hits 70%, the alarm metric should be latency or request count, not CPU.