I85PREP
⌘ K
AWS SysOps Administrator study materials
AWS SysOps Administrator study resources

Flash Sale Latency Without CPU Pressure

AWS SysOps Administrator · Put your knowledge to work in context.

All case studies

Flash Sale Latency Without CPU Pressure

A web tier runs on an Auto Scaling group (min 6, max 24, m5.large) behind an Application Load Balancer in us-east-1. Scaling is target tracking on average CPU at 60%. During a flash sale, target-group p99 response time rises from 180 ms to 2.4 s for 12 minutes while average CPU stays near 45%. The group does not scale out until minute 9; by then the ALB has returned 5% 503s. ALB access logs and EC2 metrics are enabled, and a dashboard exists that nobody trusts. RDS is not involved. The team wants faster detection and reaction without permanently doubling the fleet.

1. Which metrics and alarm thresholds would you use instead of average CPU, and why do they react earlier? How would you validate the choice?

Read the worked answer

Average CPU is the wrong signal here for two reasons:

  • It is diluted. With 6+ targets, one or two saturated instances are hidden by idle peers. Per-target metrics (RequestCountPerTarget, per-instance CPU) expose the hot shard.
  • CPU is a lagging proxy. Requests queue on connection limits, GC, or downstream calls before CPU saturates. Latency rises first; CPU never reaches the 60% target, so the alarm never fires.

Alarm on leading indicators:

  • TargetResponseTime p99 > 500 ms for 2 of 3 one-minute datapoints, aligned to the SLO rather than to CPU.
  • RequestCountPerTarget above the level the current fleet handled at 180 ms (derive from historical throughput per instance).
  • UnHealthyHostCount > 0 and TargetConnectionErrorCount as fast-fail guards.
  • If using Classic ELB metrics, SurgeQueueLength and RejectedConnectionCount are direct queueing evidence.

Validate by replaying the flash-sale window against retained CloudWatch history (15 months) and measuring how early each candidate alarm fires and how often it fires in normal traffic. Choose the threshold with the best detection-to-noise ratio, not the tightest one.

2. Propose a scaling and pre-warming design that reacts within 60 seconds while keeping steady-state cost flat. How do you prevent oscillation?

Read the worked answer

Scaling policy: move off average CPU. Use target tracking on ALBRequestCountPerTarget (a direct load measure) plus a step-scaling policy on p99 latency for fast bursts. Set instance warmup to the real boot-plus-warm time; without it, target tracking over-scales because new instances report zero load while initializing.

Pre-warming: add a scheduled action 15 minutes before known sale windows, and cap the warm pool at 1–2 instances so steady-state spend is unchanged. Predictive scaling is only useful after two weeks of repeating daily patterns; a one-off flash sale is better served by the schedule.

Oscillation control: short scale-out cooldown (about 60 s) and long scale-in cooldown (about 300 s), plus a scale-in step limit. Target tracking naturally oscillates when the metric is bursty, so prefer a metric that moves monotonically with load.

Cost neutrality: leave max at 24; the warm pool replaces, not adds to, steady-state capacity. Track cost per request before and after.

3. The team also wants to know whether the 503s were caused by the ALB or by the application. What evidence would settle it, and what remediation would you automate?

Read the worked answer

Split the evidence by layer:

  • ALB-generated 5xx typically means no healthy target, target timeouts, or connection limits. Check HTTPCode_ELB_5XX_Count, UnHealthyHostCount, and TargetConnectionErrorCount. If ELB 5xx rises while target 5xx is flat, the load balancer could not reach capacity.
  • Target-generated 5xx appears in HTTPCode_Target_5XX_Count and in ALB access logs with target_status_code populated. If those dominate, the application or its dependencies are failing.

Use ALB access logs to correlate the 12-minute window: look for target_connection_error versus application responses, and check whether unhealthy hosts were removed before or after 503s began.

Automate remediation with an EventBridge rule on the alarm state change invoking an SSM Automation document: snapshot the ASG configuration, temporarily raise the minimum capacity by a bounded amount, capture a thread dump or logs, and post to the incident channel. Keep the automated action reversible and capped so it cannot mask a deeper failure.