The Throttled Checkout Path
An e-commerce checkout service runs on 12 m7g.large instances behind an Application Load Balancer, spanning three Availability Zones. Peak traffic is 2,400 requests/minute. The team observes that p99 latency climbs from 180 ms to 1,900 ms every day between 18:00 and 20:00 UTC, while average CPU across the fleet stays at 38%. The DynamoDB orders table shows ConsumedWriteCapacity bursting to the provisioned limit during the same window, and CloudWatch reports ThrottledRequests spiking. A recent change added a synchronous write of an audit record on every checkout. The business requires p99 under 400 ms and cannot tolerate lost order writes.
1. Diagnose why average CPU is low while latency is high, and propose the highest-leverage fix.
Read the worked answer
Average CPU masks the real bottleneck. The symptom set - low mean CPU, periodic p99 spike, DynamoDB throttling aligned to the same window - points to contention on a downstream dependency, not compute saturation. The added synchronous audit write doubled write pressure on the orders table during peak, pushing it to its provisioned write limit; requests then queue on retries and exponential backoff, which inflates tail latency without loading the application CPUs.
Highest-leverage options, in order:
- Decouple the audit write - publish to a queue or stream and persist asynchronously. This removes the second write from the critical path and restores headroom. It preserves the durability requirement because the order write still completes synchronously.
- Switch the table to on-demand capacity (or raise provisioned WCU with auto scaling) so bursts are absorbed.
- Only after the above, consider instance changes - CPU is not the constraint.
Do not scale out instances: that adds more concurrent writers to an already-throttled table and can worsen the problem.
2. The team wants an alarm that fires before customers see degradation. What should it measure and why?
Read the worked answer
Alarm on customer-facing latency at the dependency boundary, not on CPU. Concretely:
- A CloudWatch alarm on the ALB
TargetResponseTimep99 with a threshold near 350 ms (below the 400 ms SLO) so there is lead time. - A DynamoDB alarm on
ThrottledRequests> 0 over a short period, plusOnlineThrottlingEventsif available, since throttling is the causal signal. - Optionally a composite alarm combining both to cut noise.
Rationale: CPU-based alarms would never fire here because CPU is not the bottleneck. Alarms should reflect either the user-visible symptom (latency) or the immediate cause (throttling), because those are the signals that move first. Use TreatMissingData: notBreaching carefully and set evaluation periods short enough (e.g., 2 of 3 datapoints at 1-minute resolution) to catch a 2-hour window early rather than after it has fully developed.