Right-Sizing an ECS Fleet After a Noisy-Neighbor Incident
A web application runs on 12 ECS instances behind a load balancer: 8 general-purpose instances (4 vCPU, 16 GiB) for the app tier and 4 memory-optimized instances (8 vCPU, 64 GiB) for a caching tier. Monitoring shows the app tier averaging 18% CPU with brief spikes to 70%, while the cache tier averages 55% CPU and 80% memory. Last month a batch reporting job was moved onto two of the app-tier instances and users reported latency spikes during report runs. The batch job is single-threaded and CPU-heavy for about 40 minutes each night. The team wants to cut instance count by 25% and asks whether to consolidate onto fewer, larger instances or keep the current shape.
1. Diagnose the latency incident and recommend how to isolate the batch workload using ECS placement and instance-type choices.
Read the worked answer
Diagnosis. The reporting job is CPU-heavy and single-threaded. On a 4 vCPU instance it can consume roughly a full core for 40 minutes, which competes with request-handling threads for CPU and for the shared network/disk path. The app tier's normal load is low (18% average), so headroom looked ample - but average utilization hides burst contention. The symptom (user latency only during report runs, only on two instances) points to co-location, not capacity shortage.
Recommendation.
- Separate the batch tier. Move reporting onto its own dedicated instances or an isolated placement group so it never shares a host with latency-sensitive app servers. If the platform supports it, use a distinct instance type sized for single-thread performance rather than many cores - a single-threaded job gains nothing from 16 vCPU.
- Use anti-affinity for the app tier so no two app instances land on the same physical host, protecting against host-level contention.
- Consider a scheduled/spot batch instance that runs only during the nightly window, since the job is predictable and interruption-tolerant with checkpointing.
- Re-baseline alarms on p95 latency and CPU steal/contention metrics, not just average CPU, so this class of incident is caught earlier.
Consolidating the app tier onto fewer larger instances would worsen the blast radius: fewer hosts means the batch job's contention affects a larger share of user traffic.
2. The team still wants a 25% reduction in instance count. Assess whether consolidation is safe for the app and cache tiers, and state the conditions under which you would approve it.
Read the worked answer
App tier: conditionally safe. 12 → 9 instances means 8 app instances become 6. Average CPU is 18%, so aggregate headroom exists, but the decision must be made on peak and on failure tolerance, not average. Check: with 6 instances, if one fails, can the remaining 5 absorb the 70% spike without saturation? If peak CPU per instance would exceed roughly 70–75% at N-1, do not consolidate. Also confirm the load balancer's health-check and connection-draining behavior so removing instances does not drop in-flight requests.
Cache tier: do not consolidate on count alone. The cache tier is memory-bound (80% memory) and already at 55% CPU. Reducing 4 → 3 memory-optimized instances cuts aggregate memory by 25%, pushing memory past safe limits and risking eviction storms that degrade the app more than the saved cost. If count must drop, move to a larger memory-optimized shape that preserves total memory, or offload cache to a managed caching service.
Conditions for approval:
- Load test at projected peak plus N-1 failure.
- Batch workload fully isolated first (otherwise the 25% cut compounds the original incident).
- Autoscaling policy in place so the fleet can grow back within minutes if forecasts are wrong.
- Roll back plan and a two-week observation window on p95 latency and error rate.
The judgement being tested: instance-count reduction is a capacity-and-resilience decision, and different tiers have different binding constraints (CPU vs memory).