Right-Sizing an ECS Fleet After a Failed Scale-Out
A web tier runs 12 ECS instances behind an SLB. Each instance is ecs.g6.2xlarge (8 vCPU, 32 GiB). CloudMonitor shows average CPU at 22% but p99 CPU touching 95% every day between 19:00 and 21:00. During that window the SLB healthy-host count drops from 12 to 9 and request latency triples. Auto Scaling is configured with a target-tracking rule at 60% average CPU and a minimum of 12 instances, but it never fires during the spike. The team wants to cut cost and has asked whether switching to 6 instances of ecs.g6.4xlarge would be safer.
1. Diagnose why the target-tracking scaling rule never fired, and explain what the p99-versus-average gap tells you about the workload.
Read the worked answer
The scaling rule keys on average CPU, which stays near 22% even while a subset of instances is saturated. Averages hide skewed load: if traffic is unevenly distributed across backends (sticky sessions, long-lived connections, or uneven SLB weighting), a few instances hit 95% while the rest idle, so the fleet average never approaches 60%.
The drop from 12 to 9 healthy hosts during the spike suggests the saturated instances are failing health checks or timing out, which removes capacity exactly when it is needed. So the rule is both measuring the wrong signal and reacting too late.
Fixes to consider: scale on a signal that reflects the bottleneck (SLB unhealthy host count, p99 latency, or per-instance CPU maximum rather than average), add a scheduled scaling rule covering the 19:00-21:00 window, and investigate why load is uneven before changing instance size.
2. Evaluate the proposal to replace 12 x 2xlarge with 6 x 4xlarge. Is the vCPU and memory math equivalent, and what risks does consolidation introduce?
Read the worked answer
The arithmetic is equivalent on paper: 12 x 8 = 96 vCPU and 12 x 32 = 384 GiB versus 6 x 16 = 96 vCPU and 6 x 64 = 384 GiB. But equivalence in aggregate does not mean equivalence in behavior.
Risks of consolidation:
- Blast radius. Losing one of 6 instances removes 1/6 of capacity instead of 1/12. During the 19:00-21:00 window you already lose 3 of 12; losing 3 of 6 would be catastrophic.
- Scaling granularity. Auto Scaling adds or removes 16 vCPU at a time instead of 8, so the fleet oscillates more and costs more per scaling step.
- Skew is not fixed. If the real problem is uneven load distribution, fewer, larger instances concentrate the same skew and may make p99 worse.
- Fewer failure domains for the same traffic.
A safer path: fix load distribution and scaling signals first, then consider a mixed fleet (for example, a baseline of larger instances plus a smaller-instance scaling group) rather than a straight swap.
3. The team also wants to cut cost by purchasing a 1-year subscription for the whole fleet. What would you advise given the observed load pattern?
Read the worked answer
The load is stable at a low baseline (about 22% average) with a predictable daily peak, which is a good fit for a mixed purchasing model rather than committing the entire fleet:
- Commit the steady baseline to subscription or savings-plan pricing. The baseline appears to be well under the current 12-instance footprint, so committing all 12 locks in capacity you may not need after right-sizing.
- Keep the burst capacity on pay-as-you-go so the scaling group can flex during the 19:00-21:00 peak without stranding prepaid capacity.
- Right-size and fix the scaling signal before signing a 1-year commitment; committing first would freeze the current over-provisioned shape.
Also confirm whether the peak is genuinely daily and predictable. If so, scheduled scaling plus a reserved baseline is cheaper and more reliable than reactive target tracking alone.