Cost and performance tuning for a batch image-processing workload
A media company processes uploaded images in batches. Jobs arrive in bursts (up to 5,000 images at once), each image takes about 20 seconds of CPU, and results are stored in OSS. The current design runs a fixed fleet of 30 ECS instances 24/7 to be ready for bursts, but they are idle most of the day. The team wants to cut cost while keeping burst throughput. They are considering spot/preemptible instances, an Auto Scaling group, and a message queue to buffer jobs. Some jobs are latency-sensitive (a small fraction), while most can wait.
1. Propose a cost-optimized architecture that preserves burst throughput, and explain the role of each component.
Read the worked answer
The core problem is paying for idle capacity to absorb unpredictable bursts. Decouple job arrival from processing and scale on demand.
Components:
- Message queue as a buffer. Uploads enqueue job messages; the queue absorbs the 5,000-image burst so producers are not blocked and workers pull at their own pace.
- Auto Scaling group of workers. Scale the worker fleet on queue depth (or a similar backlog metric) rather than running 30 instances 24/7. When the queue drains, scale in to a small baseline.
- Spot/preemptible instances for the bulk. Most jobs can wait and tolerate interruption, so run the majority of workers on spot capacity at a large discount, with checkpointing so an interrupted job is retried. Keep a small on-demand baseline for reliability and for the latency-sensitive jobs.
- OSS for results, already in place.
This converts fixed 24/7 cost into pay-for-what-you-use, while the queue smooths bursts.
2. How should the team separate latency-sensitive jobs from batch jobs, and what risks does the spot-instance choice introduce?
Read the worked answer
Separating latency-sensitive work: use a separate queue and a separate worker pool for latency-sensitive jobs, backed by on-demand instances so they are not delayed by backlog or by spot interruptions. Batch jobs go to the spot-backed pool. Routing at enqueue time (by job type) keeps the two classes from competing.
Risks of spot instances:
- Interruption. Spot capacity can be reclaimed with short notice. Mitigate with checkpointing, idempotent processing, and automatic retry from the queue so a reclaimed instance's job is reprocessed, not lost.
- Capacity variability. Spot availability fluctuates; keep an on-demand baseline and diversify instance types/zones so the fleet can still make progress during shortages.
- Throughput dips during bursts if spot is scarce - the queue absorbs the backlog, but completion time may stretch. That is acceptable for batch jobs, which is why they are the right class for spot.
Keeping latency-sensitive work on on-demand and making batch work interruption-tolerant is what makes the spot choice safe.