Right-sizing an order-processing fleet under bursty load
A retail team runs its order API on 12 ECS instances behind an SLB. Each instance is a general-purpose type with 4 vCPU and 16 GiB. CPU averages 18% over a day but spikes to 92% for 10 minutes during a flash sale every Friday, and the team has been paged for latency during three of the last four sales. The instances run a stateless Java service that takes about 4 minutes to boot and warm its JIT. The team wants to cut cost and stop the paging. They propose halving the fleet to 6 instances of the same type and enabling auto scaling to add instances during the sale.
1. Evaluate the proposed plan. What will happen during the flash sale, and what should the team change?
Read the worked answer
The plan conflates two different signals. Average CPU of 18% says the fleet is over-provisioned for baseline, so reducing to 6 is defensible for cost. But the sale spike is a capacity problem, not an average problem: if 12 instances hit 92% at peak, 6 instances cannot absorb the same load, and auto scaling will trigger only after latency has already degraded.
The bigger flaw is timing. A 4-minute boot plus JIT warm-up means any instance added at the start of a 10-minute spike is useful for at most half of it, and the scaling trigger (CPU-based) fires after the spike has begun. The team should:
- Right-size the baseline fleet using observed peak-per-instance throughput, not daily average CPU.
- Scale on a leading signal (scheduled scaling before the known Friday window, or queue depth / request rate) rather than reactive CPU.
- Keep warm capacity: pre-scale ahead of the sale and scale in after, accepting a short period of higher cost.
- Verify the SLB health check and connection draining so scale-in does not drop in-flight orders. If they still want to cut baseline, they should do it after confirming the peak-throughput math, not before.
2. The team also wants to move the service to a smaller instance family to save more. What factors decide whether that is safe?
Read the worked answer
Instance family choice is about the resource profile the workload actually needs, not just vCPU count. The team should check:
- CPU-to-memory ratio: if the JVM heap plus off-heap buffers approach 16 GiB, a memory-light family will OOM or GC-thrash regardless of vCPU.
- Sustained vs burst CPU: if the service needs full cores during the sale, a burstable family that earns credits at baseline will run out of credits exactly when load spikes.
- Network and disk throughput ceilings: per-instance bandwidth and packet rate scale with instance size; a smaller instance can bottleneck the SLB path before CPU saturates.
- Boot and warm-up time: smaller instances may take longer to warm, worsening the scaling lag identified above. The safe path is to benchmark one candidate instance under a synthetic sale-level load, confirm headroom on CPU, memory, and network, then roll it out gradually behind the SLB with health checks, keeping the old group available for rollback.