Right-Sizing a Training Job Under a Budget Cap
A team must fine-tune a 7-billion-parameter language model for document classification. They have 40,000 labeled examples averaging 900 tokens, a hard budget of 300 USD, and a two-week deadline. Their first attempt uses a single ml.p4d.24xlarge instance with batch size 2 and no distributed strategy; it is projected to take 11 days and cost roughly 900 USD. GPU utilization sampled during the run averages 22 percent, and the data loader is single-threaded reading small JSON files from S3. They cannot change the model architecture. They need a defensible plan that hits budget and deadline without silently degrading final accuracy.
1. Explain why the current configuration wastes money, and propose a concrete combination of changes to reach the budget and deadline.
Read the worked answer
Why it wastes money. At 22 percent GPU utilization the expensive accelerator is idle most of the time. Two compounding causes: (1) batch size 2 with no gradient accumulation produces tiny matrix operations that cannot saturate tensor cores, and (2) a single-threaded loader over many small S3 objects makes the GPU wait on I/O. Paying for p4d hours while the GPU idles is the dominant cost driver, not the model size.
Changes, in order of leverage:
Fix input throughput. Consolidate small JSON files into large sharded RecordIO or Parquet shards and use a multi-worker loader with prefetching. This alone typically lifts utilization well above 50 percent.
Raise effective batch size. Increase per-device batch size as memory allows and use gradient accumulation to reach an effective batch of 32–64. Larger matmuls improve throughput per step; adjust learning rate with warmup to preserve convergence.
Use distributed data parallelism across smaller instances. Several
ml.g5.12xlargenodes with SageMaker distributed data parallel often deliver better price-performance than one oversized node for this workload, and the smaller footprint reduces the cost of the idle tail.Apply parameter-efficient fine-tuning if the task permits - training a small adapter instead of all weights cuts step time and memory substantially while preserving accuracy on classification tasks.
Cap the run. Set max runtime and use SageMaker managed spot training with checkpointing to cut compute cost by a large margin, resuming from the last checkpoint after interruption.
Validate with a short pilot run measuring steps per second and projected total cost before committing the full budget.
2. The team wants early stopping but worries it will stop before the model converges. How should they configure it and prove the final model is not degraded?
Read the worked answer
Configuration. Use early stopping on validation accuracy with a patience window long enough to survive the plateau that follows learning-rate decay - typically several evaluation epochs, not one. Evaluate on a held-out split every fixed number of steps, and set a minimum step threshold so the run cannot terminate during warmup when loss is still falling steeply. Keep the best checkpoint by validation metric, not the last checkpoint, so stopping late costs nothing in quality.
Proving no degradation. Run the stopped configuration to completion on a small subsample and compare the validation curve of the early-stopped run against it; if the stopped run's best checkpoint matches the full run's best within noise, stopping is safe. Also compare the final model against the pre-change baseline on a fixed evaluation set, reporting the metric with a confidence interval rather than a point estimate.
Budget guard. Combine early stopping with a hard max-runtime so cost is bounded even if the metric keeps creeping upward. Log per-step cost so the team can see whether the marginal accuracy from additional steps justifies the spend - this turns the stopping decision into an explicit cost-benefit judgment rather than a default.