Event-Driven Order Pipeline That Must Not Lose or Duplicate Orders
An e-commerce company is building a new order pipeline. Orders arrive via API Gateway at up to 3,000 requests/second at peak. Each order must trigger inventory reservation, payment capture, and a fulfillment event, in that order. The downstream inventory service is idempotent but slow (can take 8 seconds); payment capture must never run twice for the same order. The team proposes API Gateway → Lambda → SQS → Lambda → SNS → three subscribers. They are worried about duplicate payment captures and about losing orders when a Lambda times out. They also need an audit trail of every state transition for 7 years.
1. Evaluate the proposed architecture. Where does it fail the no-duplicate-payment and no-loss requirements, and what do you change?
Read the worked answer
The proposal has two real gaps. Duplicates: SQS standard queues are at-least-once, and SNS→Lambda delivery is also at-least-once, so a retried message can trigger payment capture twice. Fix with an idempotency key on the order (client-supplied or generated at API Gateway) stored in DynamoDB with a conditional write; the payment step checks-and-sets paymentCaptured in one conditional put before calling the payment provider, so a replay is rejected. This is the only reliable defense - do not rely on exactly-once delivery claims.
Loss: a Lambda that times out after SQS's visibility timeout, or that exhausts retries into a dead-letter queue nobody watches, loses orders. Set the queue visibility timeout to at least 6× the Lambda timeout, configure a DLQ with an alarm, and make the consumer idempotent so redelivery is safe. For the slow 8-second inventory call, a synchronous Lambda chain risks timeouts - decouple with a queue per stage.
Ordering: the required sequence (inventory → payment → fulfillment) should be enforced by a state machine (Step Functions) or by chained queues with a status field, not by hoping SNS fan-out preserves order. SNS fan-out to three subscribers runs them in parallel, which violates the required order.
2. Propose the revised design and explain how the 7-year audit trail is produced without slowing the hot path.
Read the worked answer
Revised flow: API Gateway → Lambda (validate, assign idempotency key) → SQS → Step Functions workflow. The state machine calls inventory (with retry/backoff), then payment (conditional DynamoDB write as the idempotency gate, then provider call), then emits fulfillment. Each state writes a status record to DynamoDB. Failures route to a DLQ with an alarm. Use a FIFO queue keyed by order ID only if strict per-order ordering is needed; otherwise standard queue plus the state machine's explicit sequencing is sufficient and scales better at 3,000 rps.
Audit trail: emit a state-transition event to EventBridge from each state, and route all events to a Firehose delivery stream into S3 with a lifecycle policy (or S3 Glacier after 90 days) for 7-year retention. This is asynchronous, so it does not add latency to the order path; if an audit write fails, EventBridge retries and the DLQ captures it. Enable CloudTrail and S3 Object Lock in compliance mode if the auditors require immutability. Query with Athena for investigations. Keep the DynamoDB status table as the operational source of truth and S3 as the immutable archive - do not use one for both.
3. At 3,000 rps peak, what capacity and throttling concerns would you flag before launch?
Read the worked answer
Check each hop against peak, not average. API Gateway and Lambda scale, but DynamoDB needs on-demand or provisioned capacity sized for the conditional-write rate, and the idempotency table should use TTL to expire keys after a window (e.g., 30 days) so it does not grow unbounded. SQS standard queues scale nearly without limit, but FIFO queues are capped per message group - at 3,000 rps with many orders, FIFO throughput per group can become the bottleneck, another reason to prefer standard queues plus state-machine sequencing. Step Functions standard workflows have a transactions-per-second quota that must be raised via support before peak; express workflows are cheaper but have a 5-minute duration limit, which the 8-second inventory call plus retries can approach. Set Lambda reserved concurrency so the order path is not starved by other functions, and configure API Gateway throttling (account and stage level) with a usage plan so a client cannot consume the whole budget. Load-test at 2× peak with the payment provider sandbox to confirm the idempotency gate holds under contention.