Choosing Between a Managed Queue, a Managed Stream, and a Cron Job
An e-commerce team processes order events. Peak is 4,000 events/minute for about 20 minutes during flash sales; baseline is 40 events/minute. Downstream, three independent consumers must each see every event: fraud scoring (must run within 2 seconds), invoice generation (can run hours later), and an analytics archive (must retain events for 12 months and allow replay). The current design uses a single relational database table as a queue, polled by a cron job every minute, and it is falling behind during peaks. The team wants the smallest architectural change that fixes the backlog without losing events.
1. Which messaging services fit each consumer, and why is the current cron-polled table failing?
Read the worked answer
The table-as-queue fails because polling every minute caps throughput at one batch per minute, creates lock contention on the table, and gives no backpressure signal - during a 4,000/min peak the producer outruns the consumer and the backlog grows without bound. The fix is to decouple producers from consumers with a durable buffer. For fraud scoring, a queue with at-least-once delivery and a worker pool sized for peak gives low latency and simple retry semantics. For invoice generation, the same queue pattern with a slower consumer group is fine, or a separate queue so a slow invoice worker cannot starve fraud scoring. For the 12-month archive with replay, a durable log/stream that retains events and lets consumers re-read from an offset is the right fit - a standard queue deletes messages after acknowledgment and cannot replay. The judgement point is that fan-out to three independent consumers is easier with a log-style stream plus separate consumer groups, but the latency-sensitive path may still prefer a dedicated queue.
2. How should the team size capacity for the 20-minute peak without paying for it all month?
Read the worked answer
Separate buffer capacity from consumer capacity. The buffer must absorb the peak regardless of consumer speed, so it should be provisioned or configured for sustained burst with retention long enough to drain - this is cheap relative to compute. Consumer capacity should scale on backlog depth or age, not on a fixed schedule, so it grows during the flash sale and shrinks afterward. Set a maximum concurrency ceiling so a runaway consumer cannot amplify cost, and set an alarm on oldest-message age so the team learns about backlog before customers do. The trade-off to document: scaling on backlog reacts after a delay, so for a known flash-sale window a pre-warm is justified; for unknown spikes, reactive scaling plus a sufficient buffer is the cheaper choice.
3. What failure mode should the team test before launch, and what does a passing test prove?
Read the worked answer
Test consumer failure during peak: kill a consumer mid-burst and confirm no event is lost, duplicates are handled idempotently, and the backlog drains within an agreed time after recovery. Also test a poison message - one event that always fails processing - and confirm it moves to a dead-letter path instead of blocking the queue forever. A passing test proves the system tolerates at-least-once delivery and consumer crashes; it does not prove exactly-once processing, so downstream writes (invoices, fraud decisions) must be idempotent by business key. The judgement point is that the architecture's real risk is not throughput but duplicate side effects and stuck messages, which only a failure test reveals.