Interrupted Image Export on Spot ECS
A batch team runs a nightly image-rendering job on 20 ECS instances launched with the Spot (preemptible) billing method in a single zone. Each instance processes a shard of 500 files and writes partial output to the instance's local ephemeral disk before uploading a final tar to OSS at the end of the shard. Average runtime is 90 minutes, but preemption notices arrive roughly every 40 minutes across the fleet. On a recent night, 6 instances were reclaimed mid-shard; the team found no partial output in OSS and had to rerun entire shards. The team wants to keep Spot pricing but stop losing work. They cannot change the rendering code. They ask whether switching to a different zone or adding a second instance type would fix the loss.
1. Diagnose the root cause of the lost work and explain why zone or instance-type diversification alone does not solve it.
Read the worked answer
The loss is not caused by preemption itself but by where state lives. All intermediate output is on ephemeral local disk, which is destroyed on reclamation, and the job only uploads at the end of a 90-minute shard. Preemption every ~40 minutes means the job almost never reaches its upload point. Zone or instance-type diversification reduces the probability of a simultaneous fleet-wide reclaim and can give the job a better chance of finding capacity, but it does not change the fact that a reclaimed instance takes its partial work with it. The fix must make progress durable and resumable: checkpoint partial output to OSS (or a mounted NAS/cloud disk) periodically, and make the shard unit small enough that a reclaim costs minutes, not hours.
2. Recommend a concrete architecture change that keeps Spot economics and tolerates preemption, given the code cannot change.
Read the worked answer
Since the code cannot change, put the durability around it rather than inside it:
- Shrink the unit of work. Split each 500-file shard into many smaller shards (e.g., 25 files) so a reclaim loses at most one small shard.
- Externalize output. Mount an OSS-backed or NAS volume at the output path the code already writes to, so partial artifacts survive instance loss. NAS is the lower-friction option because it presents a normal filesystem.
- Handle the preemption notice. Use the instance metadata preemption signal to trigger a graceful drain: stop accepting new shards, let the current small shard finish or upload, then release.
- Make the queue authoritative. Track shard completion in a durable queue/table so a replacement instance picks up only unfinished shards.
- Diversify as a secondary measure. Multiple instance types and zones improve capacity availability, but they are a resilience add-on, not the durability mechanism.
3. The team also asks whether they should simply switch to pay-as-you-go to eliminate the problem. Give a balanced recommendation.
Read the worked answer
Pay-as-you-go removes preemption but at materially higher cost for a 20-instance nightly fleet. The better answer is to keep Spot and engineer for interruption, because the workload is already batch and idempotent-friendly. A hybrid is often best: run the bulk on Spot with checkpointing, and keep a small pay-as-you-go or reserved baseline for the tail of the job or for deadline-critical nights. The decision should be driven by the cost of a missed deadline versus the Spot discount, not by fear of preemption. If the team implements durable checkpoints and small shards, Spot becomes safe; pay-as-you-go becomes a fallback for the final stragglers only.