AWS Machine Learning Specialty (AWS): Weekly 30-Hour Training on Spot Instances That Keep Getting Interrupted

Root cause: instance-local EBS is destroyed when the instance is reclaimed, so there is nothing to resume from. Managed Spot Training restarts the job after an interruption and runs until MaxWaitTimeInSeconds; resumption only works if the checkpoint survived.

Fix:

1. Set checkpoint_s3_uri and checkpoint_local_path. The framework writes to the local path and SageMaker continuously syncs it to S3, restoring it at the start of the restarted job. Writing to /opt/ml/checkpoints without the S3 URI is the exact mistake being made here.
2. Make the framework actually use the hook. For PyTorch, save model + optimizer + scheduler + step counter via torch.save into the local checkpoint path; for a custom container, implement the save/load contract. Saving weights only (without optimizer state) restarts optimisation cold and wastes many hours.
3. Shorten the checkpoint interval to every 10-15 minutes, not every epoch. A 30-hour run with epoch-length checkpoints loses hours per interruption.
4. Set MaxWaitTimeInSeconds comfortably above MaxRunDurationInSeconds plus the expected interruption delay, so a reclaimed instance is actually replaced rather than the job silently failing.
5. Verify by deliberately interrupting a short test job and confirming the resumed run starts from the recorded step, not from step 0.

With interruptions now costing minutes instead of hours, the spot discount becomes a real saving rather than an illusion.

Treat it as a scheduling-risk decision, not a discount question.

• Interruptions are now cheap (a few minutes) but not free: after four interruptions in six runs, the probability that a given run needs at least one restart is high. If the retrain must publish by a fixed time, the tail matters, not the mean.
• Keep spot for experimentation and non-blocking runs, where a delay is harmless.
• For the release-gating run, either (a) run on-demand, or (b) run on spot with checkpointing and a deterministic fallback: if elapsed time exceeds a threshold that would threaten the deadline, launch the on-demand version immediately rather than waiting out MaxWaitTime.
• Better still, shrink the run first. 30 hours on 8 p3.16xlarge for a recommender is usually reducible: fewer instances with better utilisation, mixed precision, gradient accumulation, a smaller embedding table after pruning rarely-seen items, or distillation into a smaller student. Halving the runtime often costs less accuracy than the release cadence costs in risk.

Quantify it: expected cost of on-demand x P(success) versus spot cost plus the cost of a missed release window. The answer is usually "spot with checkpointing plus an on-demand escape hatch", not one or the other.

1. Reduce the work (largest win, do first). 40M pairs usually means every user x every item. Use a two-stage design: a cheap retrieval step (ANN index, co-visitation counts, popularity per segment) cuts candidates to 2-3M, then the expensive model scores only those. That is often a 10x reduction with a small quality cost, and it is far cheaper than any infrastructure change.

2. Benchmark single-instance throughput, then size. Run one Batch Transform job on a fixed shard, measure rows/second, compute required instance count = total rows / (throughput x 10,800 s), and add 30% headroom. Do not guess instance counts.

3. Shard the input correctly. Batch Transform parallelises across input objects: split the 40M pairs into many S3 objects (not one giant file) so instance_count x max_concurrent_transforms are all busy. Set max_payload and mini-batch size so each GPU invocation is large enough to amortise per-request overhead, and use AssembleWith to produce one usable output.

4. Right-size the hardware. Use SageMaker Neo to compile the model for the target instance (typical multi-x speedups on GPU without accuracy change) and pick a GPU with enough memory to hold the batch without OOM.

5. Only then consider scale. Add instances; spot is not available for Batch Transform, so cost control comes from the candidate reduction in step 1 and from Neo, not from purchasing options.

Monitor per-job ModelLatency and job duration each night and alarm if the runtime trends toward the window.