Queue Backlog During a Flash Sale
An order-intake pipeline runs 6 API nodes behind a load balancer, each sustaining 150 req/s at 60% CPU. Requests land on a work queue consumed by 8 workers; each worker handles one message at a time, averaging 25 ms, so the fleet drains roughly 320 msg/s. Baseline traffic is 100 msg/s. During a 12-minute flash sale the queue grew to 90,000 messages, so the team raised the worker count from 8 to 16 - throughput rose only about 12%. The database permits 200 connections total and each worker opens 20. The queue's visibility timeout is 30 s; the worker's 99th-percentile handling time under load is 45 s. Nothing else in the stack changed.
1. Identify the binding constraint. Which measurements would separate it from plain worker CPU saturation, and what do the given numbers already tell you?
Read the worked answer
Connection arithmetic settles most of it. 8 workers × 20 connections = 160 of the 200 allowed. Doubling to 16 workers asks for 320, which is 60% more than the database will grant. The second eight workers therefore spend most of their time blocked on pool acquisition, so you get single-digit-to-teens percentage improvement instead of ~2x. The observed 12% is exactly the shape of a connection-starved scale-out.
The visibility timeout inflates the work. A 30 s timeout against a 45 s p99 means messages become visible again while still being processed. Each redelivery is duplicate effort, so useful throughput is below the nominal 320/s. That also explains the growth rate: 90,000 messages in 12 min is a net accumulation of about 125 msg/s, consistent with arrivals near 350–400/s against an effective drain well under that.
Distinguishing measurements:
- Worker CPU flat or falling while queue depth climbs → not CPU-bound.
- Active connections pinned at the cap, plus rising p99 connection-acquisition wait → pool starvation confirmed.
- Redelivery / duplicate-suppression counters climbing → timeout is too short.
- Database CPU elevated but with lock waits and I/O, not raw core saturation → the DB is queueing, not computing.
Conclusion: the constraint is the connection budget plus duplicated work, not worker CPU. Scaling workers was the wrong lever for the wrong reason.
2. Give an ordered remediation plan for the next flash sale, and state what you would refuse to do.
Read the worked answer
Order matters because premature scaling multiplies duplicate work.
- Fix correctness first. Raise the visibility timeout to at least 2–3× p99 (e.g., 120 s) and make handlers idempotent on a message ID. Until redelivery stops creating duplicate side effects, every throughput number you measure is inflated by wasted work.
- Budget concurrency centrally, not per worker. Move to a shared pool capped around 150 connections (75% of 200), with a bounded queue in front. Now 16 workers share one budget instead of each grabbing 20.
- Cut round trips per message. Batch or combine writes so a message costs fewer statements. This raises throughput at a fixed connection count - the only lever that doesn't trade against the database.
- Only then add workers, and autoscale on oldest-message age rather than depth, because depth scales with arrival rate and says nothing about drain.
What I would refuse: raising the database's connection limit mid-incident. It converts a bounded queueing problem into memory and lock contention that is far harder to diagnose, and it hides the real bottleneck. Equally, I would refuse to shorten the visibility timeout or drop idempotency to 'catch up' - that buys throughput by corrupting order state.
3. What single alert best captures this failure mode, and why is queue depth a poor choice?
Read the worked answer
Alert on oldest-message age (consumer lag in seconds) against an SLO: warn at 60 s, page at 5 min.
Why age and not depth: depth is meaningless without a drain rate. 90,000 messages is a routine afternoon at a higher steady-state rate and an emergency at this one, so a fixed depth threshold either pages constantly or never. Age converts depth and drain into the one quantity users feel - how stale is the oldest unprocessed order.
Pair it with two cause signals rather than more symptom alerts:
- Redelivery ratio > 1% - the leading indicator that the visibility timeout has been overtaken.
- Connection-acquisition p99 - tells you whether you are starving on the database before the lag SLO fires.
That combination pages once, names the likely cause, and works at any traffic level.