Sizing a Queue Consumer Before a Seasonal Peak
An order-processing service pulls messages from a queue and calls a downstream payment API. Current steady-state load is 40 messages per second. Each message triggers exactly one payment call, which has a p95 latency of 300 ms and a hard timeout of 2 s. The consumer runs 8 worker threads, each handling one message at a time. Last year's peak was 3.5x steady state and lasted about 90 minutes. The queue has no dead-letter handling and messages are deleted on read. Product wants no order to wait more than 60 seconds at peak. The payment provider enforces a rate limit of 120 calls per second and returns 429 errors above that, which the consumer does not currently retry.
1. Calculate whether 8 workers can sustain last year's peak, and state the binding constraint.
Read the worked answer
Throughput per worker is 1 / latency. At p95 latency of 0.3 s, one worker completes about 3.33 messages per second. Eight workers give about 26.7 messages per second at p95.
Peak demand is 40 x 3.5 = 140 messages per second. So 8 workers sustain roughly 19% of peak. Even at the optimistic assumption that every call finishes at the median rather than p95, the workers cannot close a 140 vs ~27 gap.
Workers needed at p95: 140 / 3.33 = 42 workers. That exceeds the provider's 120 calls per second limit, so the binding constraint is not worker count but the downstream rate limit. At 120 calls per second the system is capped at 120 messages per second no matter how many workers exist, which is still below the 140 peak.
So there are two separate problems: concurrency is far too low for the peak, and even correct concurrency would hit an external ceiling. The design must both raise worker count toward ~42 and add a client-side rate limiter plus a backlog strategy, because the provider ceiling cannot be raised by adding capacity on our side.
2. Given the 60-second wait target and the rate limit, propose a concrete plan and identify what must change in the queue semantics.
Read the worked answer
Work backward from the target. At a 120 calls/second ceiling, a 90-minute peak can absorb 120 x 5,400 = 648,000 calls. Demand at 140/second for 5,400 seconds is 756,000. The excess is 108,000 messages, which must either be smoothed or shed. A 108,000-message backlog drained at 120/second while new traffic continues at 140/second never drains, so smoothing alone fails; the plan must reduce demand or raise the ceiling.
Concrete plan:
- Negotiate a temporary rate increase with the provider, or shard across a second provider account. This is the only way to serve 140/second.
- If no increase is possible, apply admission control: prioritize high-value orders and defer or reject the rest with an explicit customer-facing message. A 60-second wait target is unachievable for all traffic at 140/second against a 120/second ceiling.
- Size workers to ~42 at p95, add a token-bucket limiter set below 120 to leave headroom, and retry 429s with exponential backoff and jitter.
Queue semantics changes: messages are currently deleted on read, so a crash loses orders. Switch to explicit acknowledgement after the payment call succeeds, add a visibility timeout longer than the 2 s call timeout, and add a dead-letter queue for repeated failures. Without these, retries and backpressure cannot be implemented safely.