Duplicated Orders in an SQS-Driven Importer
An order service runs a Node.js Lambda behind an SQS standard queue. The event source mapping uses batch size 10, the queue's visibility timeout is 30 seconds, and the function timeout is 120 seconds. No maxReceiveCount is configured and there is no dead-letter queue. Under a seasonal peak the backlog climbs past 200,000 messages and CloudWatch shows iterator age growing steadily. About 4% of orders are written to DynamoDB twice, yet the function's ConditionalCheckFailedException metric is flat at zero. Support also reports one malformed message that keeps reappearing for hours, and every time it reappears the other nine messages in its batch are processed again. The team wants the queue drained without manual intervention and without losing messages.
1. Orders are being duplicated even though the function calls PutItem for every message. What is producing the duplicates, and which two changes actually stop them?
Read the worked answer
Mechanism 1 - visibility timeout shorter than the work. The message becomes visible again 30 seconds after receipt while the function may still be running for up to 120 seconds. A second consumer picks it up, both finish, both write. Any redelivery (throttling, timeout, batch failure) has the same effect.
Mechanism 2 - no idempotency. ConditionalCheckFailedException sitting at zero is the giveaway: the writes are unconditional PutItem calls, so a duplicate message silently overwrites the first write and no one notices until downstream systems act twice (double shipment, double email).
Fix A: raise the visibility timeout above the function timeout - AWS recommends at least 6× the function timeout, so 720 seconds for a 120-second function - or reduce the function timeout. This shrinks the window but cannot eliminate it, because at-least-once delivery means duplicates are always possible.
Fix B (the real fix): write conditionally on a business idempotency key derived from the message (for example attribute_not_exists(orderId)), and treat the condition failure as a successful no-op. Add a separate idempotency record with a TTL if you need to deduplicate work that happens before the DynamoDB write. Do the idempotent write first, before any non-idempotent side effect.
2. One malformed message is stuck in a loop and drags nine innocent messages with it on every retry. How should the consumer and the queue be reconfigured so this self-heals?
Read the worked answer
Stop failing the whole batch. By default Lambda treats any thrown error as a batch-level failure: nothing is deleted, and all ten messages return to the queue. Enable partial batch responses: set FunctionResponseTypes: ["ReportBatchItemFailures"] on the event source mapping and have the handler return {"batchItemFailures":[{"itemIdentifier":"<messageId>"}]} listing only the records that failed. Successful records are deleted. This is the single change that removes the collateral damage.
Stop the poison message from looping forever. Attach a redrive policy to the source queue with a maxReceiveCount of roughly 3–5 and target a dead-letter queue. The malformed message is then quarantined instead of cycling for hours.
Close the loop operationally. Alarm on ApproximateNumberOfMessagesVisible on the DLQ, and keep a redrive path back to the source queue once the bug is fixed. Also validate the payload at the top of the handler and classify errors: a permanent validation failure should be reported as a batch item failure immediately rather than retried, while transient errors (throttling, downstream 5xx) should be retried. Without this classification you will keep burning visibility-timeout cycles on messages that can never succeed.
3. The backlog is 200,000 messages and iterator age keeps rising. What is the right way to scale the consumer, and what can go wrong?
Read the worked answer
Scaling lever: Lambda's SQS event source mapping scales by adding pollers, up to 1,000 concurrent invocations per queue for a standard queue (and there is a per-queue maximum on how fast new pollers are added). Increasing batch size and adding a batch window reduces invocation count and per-message overhead, which usually buys more throughput than any handler micro-optimization.
What goes wrong:
- Account concurrency is shared. The default is 1,000 concurrent executions across the whole account. If this consumer saturates it, every other function in the account is throttled - the classic cause of unrelated 429s and Lambda
Throttlesalarms firing during an incident. Set reserved concurrency on this consumer to cap its blast radius, understanding that reserved concurrency is subtracted from the unreserved pool. - Downstream is the real ceiling. Fanning out to 1,000 concurrent writers will push DynamoDB into throttling, and SDK retries then multiply latency. Confirm the table has headroom (or on-demand mode) before scaling the consumer, and consider a token bucket or a lower batch size for write-heavy handlers.
- Reserved concurrency set too low starves the consumer permanently; it caps throughput rather than guaranteeing it. Size it from measured per-invocation duration and the target drain rate, not by guesswork.
Do not simply raise the function timeout: longer invocations hold concurrency slots and worsen iterator age under a fixed concurrency budget. Draining speed is concurrency × batch size ÷ duration.