OOM and Low Utilization in a 13B Fine-Tuning Job
You run a 13B-parameter LLM fine-tuning job on eight A100 80GB GPUs using PyTorch FSDP, bf16, sequence length 4096, micro-batch size 2, and gradient accumulation 16. Activation checkpointing is disabled. At step 300, one rank raises CUDA out of memory; nvidia-smi shows 79.8 GB used on every GPU. torch.cuda.memory_summary() on the failing rank shows 12 GB reserved but unallocated and many small fragmented blocks. The data loader processes are at 100% CPU and the GPUs idle about 35% of the time. The team wants to raise global batch size from 256 to 512 for convergence and suggests either buying more GPUs or enabling MIG on the A100s. Your SLO is to finish the current run in 10 days.
1. How do you distinguish a true capacity shortfall from allocator fragmentation and a data-pipeline bottleneck?
Read the worked answer
The evidence points to a mixed problem, not a simple shortage of GPU memory. A 12 GB reserved-but-unallocated block means the caching allocator is holding memory it cannot reuse, often because variable sequence lengths or evaluation/checkpoint code created many small blocks. The failure at step 300, after hundreds of stable steps, is more consistent with fragmentation or a transient allocation than with a model that was always too large. At the same time, 100% CPU in the data loader and 35% GPU idle means the job is not compute-bound; adding GPUs would raise cost without fixing stalls. Use torch.cuda.memory_allocated() versus memory_reserved(), enable allocator stats, and log max sequence length per batch. Check whether the OOM coincides with checkpoint saving, evaluation, or a long-sequence batch. For the data path, measure samples per second, worker queue depth, and host-to-device copy time. A true capacity shortfall would show allocated memory near the limit on every rank with no large reserved-unallocated pool. Here, the correct first move is allocator and data-pipeline remediation, not procurement.
2. Evaluate the proposals: more GPUs, MIG on A100s, activation checkpointing, and larger global batch. What would you actually do?
Read the worked answer
More GPUs would add memory and compute, but procurement and cluster integration will not fit a 10-day SLO, and it does not address the 35% idle time. MIG on A100 80GB is a poor fit: MIG partitions memory and SMs into smaller slices, which reduces per-process memory rather than increasing it, and a 13B FSDP job needs the full GPU memory pool. It may help inference consolidation, not this training run. Activation checkpointing is the right memory lever: enabling it reduces activation memory substantially at the cost of recomputation, which is acceptable because the GPUs are currently idle 35% of the time. For global batch 512, do not raise micro-batch from 2; instead increase gradient accumulation from 16 to 32 after memory stabilizes, or use a sequence-length-aware curriculum. Also set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation, call empty_cache() only at safe boundaries, and fix the data loader with more workers, prefetching, and pre-tokenized shards. The combination addresses memory, utilization, and convergence batch size without buying hardware.
3. What is your 10-day execution plan, including validation criteria before committing the full run?
Read the worked answer
Day 1–2: instrument memory and data throughput, enable expandable segments, remove unnecessary per-step allocations, and pre-tokenize or cache the dataset so the loader can sustain the target samples per second. Day 3: enable activation checkpointing and run a 500-step canary with the same sequence distribution, watching allocated versus reserved memory and GPU idle time. Day 3–4: fix the loader bottleneck, then test gradient accumulation 32 for global batch 512; confirm loss curves and optimizer step counts match the intended schedule. Day 5: run a longer 2,000-step validation with checkpoint save and evaluation included, because those are common OOM triggers. Acceptance criteria: no rank above 90% of HBM with fragmentation headroom, GPU idle below 10%, stable NCCL and data-loader throughput, and no OOM across checkpoint and eval cycles. Only then commit the full run. If the canary still OOMs, reduce sequence length or use selective activation checkpointing rather than reverting to a smaller global batch, since convergence is a stated requirement.