A Queue Backing Up for Nine Hours While a Server Idles at 6% CPU

The workload is a classic fan-out batch job: stateless per-item work, bursty arrival, latency tolerance measured in minutes.

• S3 event notification → SQS queue (with a dead-letter queue). Decouples arrival from processing and absorbs the 300,000-image burst as a buffer instead of a backlog with no back-pressure. The DLQ is what turns silent failures into visible ones.
• Consumers: functions or containerised tasks. Thumbnail generation is short and CPU-bound; a container-image function with ImageMagick baked in fits, subject to the 15-minute execution ceiling and the concurrency quota, which must be checked against the burst shape before committing. If any image type risks exceeding the ceiling, use a containerised worker on a serverless compute service instead - same skills, no servers.
• Scale from queue depth, not from a schedule. This is the actual fix for the 9-hour wait: capacity follows backlog instead of sitting at 6%.
• Idempotency store. A small key-value table recording processed (source version, output size) pairs, written conditionally before the output is published.

Justify by eliminating the two failure modes - idle capacity and unbounded backlog - without adding operations the team will not staff.

Root cause: at-least-once delivery meeting a non-idempotent writer. Queue-based and event-driven systems guarantee delivery at least once; retries, visibility timeouts expiring mid-job, and redrives all cause the same message to be processed twice. Writing to a key derived purely from the source filename means the second attempt silently overwrites the first - including a human-edited file.

Fix in three parts:

1. Make the output key a function of the input version, not just the name - bind it to the source object version or content hash, so a reprocess of the same input is a no-op rather than an overwrite.
2. Record processed units in a conditional write - attempt the marker insert, proceed only if it succeeded. This makes the handler idempotent under concurrency, not just under retries.
3. Separate machine output from human output by prefix. Generated thumbnails go to a machine-owned prefix; curated files live elsewhere and are never a write target. That is a policy fix, and it is the one that prevents recurrence when a new pipeline is written next year.

Also set the queue visibility timeout above the maximum job duration, or you will manufacture duplicate work.

Rough model: 4M images × 3 sizes = 12M executions per month. At an assumed 1 second and 1 GB of memory each, that is 12M GB-seconds and 12M requests. At published on-demand rates (approximately $0.0000166667 per GB-second and $0.20 per million requests) that lands near $200 compute plus a couple of dollars for requests - under the $450.

But the estimate is soft in one direction: function cost scales with memory × duration, so 2 GB or 3 seconds doubles or triples it. Tune memory against duration - higher memory often cuts wall-clock time more than it raises the rate, and the cheapest configuration is rarely the smallest one. Add storage, queue, and logging costs before declaring victory.

Why cost is not the question: the current spend is modest and the stated pain is a 9-hour backlog and a correctness bug. The proposal should be justified on burst elasticity and on idempotent, retry-safe processing - then cost becomes a bonus. A design that solves the backlog for $600 is a better answer than one that saves $250 and still leaves images waiting, and the reverse framing invites a rejection the architecture does not deserve.