Why 64 GPUs Ran Only 1.25x Faster Than 32
A 7B dense model trains at 44% MFU on 32 A100s (4 nodes) with a 1.00 s step time. The team scales to 64 GPUs (8 nodes) over 200 Gb/s InfiniBand, holding per-GPU micro-batch and sequence length constant, so per-GPU work per step is unchanged. Step time becomes 1.60 s and MFU drops to 33%. During the run, a full checkpoint is written every 30 minutes to a shared NFS mount, blocking all ranks for 5.5 minutes. The training job also uses 64 dataloader workers pinned to the same host CPUs as the GPU feeder threads, and host CPU utilization sits at 92% when the job is not checkpointing. Leadership wants to go straight to 128 GPUs to recover the schedule.
1. Quantify the scaling loss and separate how much of the MFU drop is attributable to what. What would you measure before touching the model code?
Read the worked answer
With per-GPU work constant, doubling GPUs should keep step time flat at 1.00 s. Actual is 1.60 s, so throughput scales as 2 / 1.6 = 1.25x for 2x the hardware: 62.5% scaling efficiency. MFU 44% -> 33% is a ratio of 0.75, consistent with the same story.
Attribution before blame:
- Checkpointing is 5.5 / 30 = 18% of wall-clock, and it is synchronous - every rank stalls. That alone explains most of the gap: it is larger than the entire efficiency deficit.
- Once you remove 18% of stall from a 1.60 s step, the compute-only step is roughly 1.31 s against an ideal 1.00 s - still a 31% comms/exposure overhead, which is plausible for a step whose per-GPU work is now small relative to the all-reduce volume.
- Do not assume network: 200 Gb/s IB with the same model and topology should not produce 31% overhead on its own at this size, but the ratio of communication to compute per step is what changed, not the link.
Measure, in order: (a) time split between forward/backward/optimizer/all-reduce/checkpoint-write/input-wait, logged per rank over ~50 steps; (b) whether the input-wait gap is nonzero given 92% host CPU - with dataloader workers competing with feeder threads, input stalls will masquerade as network stalls if you only look at the collective timers; (c) the straggler distribution across ranks, since a single slow node drags every collective.
2. Propose the concrete fixes, in order, with the expected recovery for each.
Read the worked answer
1. Make checkpointing non-blocking (biggest, cheapest). Write sharded checkpoints to local NVMe per node and copy to NFS asynchronously; keep only a small metadata write synchronous. Expect to recover most of the 18% wall-clock overhead - roughly 1.60 s -> ~1.31 s of effective step time, i.e. MFU back toward the high 30s before any network work.
2. Fix the host CPU contention. Give dataloader workers and GPU feeder threads separate cores, reduce worker count to match actual throughput, and pre-stage shards. At 92% CPU this is a real stall source today, and it will get worse at 128 GPUs where each rank has less work to hide the gap behind.
3. Then look at communication. With per-GPU work halved relative to the 32-GPU config, collective overhead is a larger fraction of the step. Options: overlap gradient reduction with backward, tune bucket sizes, ensure the topology mapping puts peers on the same leaf switch, and confirm NCCL is using the intended interfaces rather than a management NIC.
4. Re-baseline before scaling. Recompute MFU and scaling efficiency at 64 GPUs after (1)-(3). The 128-GPU decision should be made on the corrected curve.
3. Leadership asks whether to book 128 GPUs for next week. What is your recommendation and the reasoning you give?
Read the worked answer
Do not book 128 yet - fix 64 first.
The current 62.5% efficiency is not a property of scale; it is dominated by an 18% synchronous checkpoint stall and host CPU contention, both of which are configuration bugs. Buying more GPUs multiplies the bill while leaving the bug in place. At 64 GPUs the 1.25x return already means you are paying for 2x hardware to get 1.25x throughput.
Scaling further also makes the physics worse before it gets better: per-GPU work per step keeps shrinking, so communication overhead as a fraction of the step rises, and any input-side stall that was hidden by compute becomes exposed. Two of the three fixes above (checkpoint, dataloader) are pure configuration and cost GPU-hours near zero.
The recommendation to give: spend one to two days on (1) and (2), re-baseline MFU at 64 GPUs, and only then run a short 128-GPU canary of a few hundred steps to measure the new curve. Book the extra capacity against that measurement, not against the schedule pressure - the schedule is already lost, and adding unreliable throughput makes the estimate worse, not better.