Migrating an 8-Card ResNet-50 Job to Ascend 910 on ModelArts
A retail team is moving a PyTorch ResNet-50 training job from an on-premises 8×V100 server to a ModelArts training job using one node with 8 Ascend 910 NPUs. The dataset is 2.1 million JPEG images (about 280 GB) stored in an OBS bucket, and the job must finish within the platform's 3-day limit. On the GPU server, batch size 256 per card in FP32 converges in 18 hours. On ModelArts, the first attempt at batch size 256 per card fails at step 0 with an NPU out-of-memory error; batch size 64 runs but delivers only about 310 images/s, and each epoch re-reads the full dataset from OBS. The loss curve at batch 64 also diverges from the GPU baseline. The engineers propose simply reducing the batch size to 32 and increasing the number of epochs. The model code currently calls .cuda() and uses torch.cuda.amp. Changing the network architecture is out of scope.
1. Identify the two independent causes of the low throughput and the OOM, and state which one you would fix first. Justify the order.
Read the worked answer
The two causes are largely independent:
- Throughput (~310 images/s): the job re-reads JPEGs from OBS every epoch, so the bottleneck is object-storage round trips plus CPU JPEG decode, not NPU compute. Evidence: throughput is far below the GPU baseline and the symptom repeats each epoch.
- OOM at batch 256: the run is FP32 with per-card batch 256, and
.cuda()/torch.cuda.ampdo not activate Ascend mixed precision. Activation memory exceeds NPU capacity at that batch size.
Fix the data path first. It dominates wall-clock time, does not change the optimization problem, and can be validated in minutes. Concretely: copy the dataset once to the job's local /cache disk or a cached OBS path, convert JPEGs to a sequential binary format (RecordIO/TFRecord), and decode in parallel inside the data pipeline. Then enable mixed precision with loss scaling through the torch_npu AMP API and re-test batch 256. Only if memory still fails should you use gradient accumulation; reducing to batch 32 changes batch-norm statistics and convergence and should not be the first move.
2. The team wants an effective batch size of 2048 across 8 cards. Compare (a) batch 32 per card with 8-step gradient accumulation and (b) batch 256 per card with mixed precision. Which do you recommend, and what must be verified?
Read the worked answer
Both give an effective batch of 2048, but they are not equivalent.
(b) batch 256 per card with mixed precision is preferable if it fits. It preserves per-card batch-norm statistics, keeps one optimizer step per data batch, and avoids accumulation code and checkpoint complexity. The risk is numerical: loss scaling must be tuned and overflow checks enabled.
(a) accumulation reduces peak memory but changes batch-norm behavior: each micro-batch computes statistics over 32 samples, not 256. If batch norm is used, you must either keep batch 256 for BN layers, switch to sync BN, or recalibrate. It also slows the step loop and complicates gradient clipping.
Verification for either route:
- Run 200–500 steps and compare the loss curve against the GPU baseline at the same effective batch; divergence at batch 64 earlier suggests a learning-rate or precision issue, not just batch size.
- Confirm no implicit FP32 fallback and that HCCL collectives initialize across all 8 cards.
- Scale the learning rate according to the chosen effective batch and check warmup.
- Measure images/s end to end, including data loading, not just NPU utilization.
3. What evidence would you require before committing the full 3-day production run?
Read the worked answer
A short, instrumented qualification run is cheaper than discovering a failure on day 2. I would require:
- Throughput evidence: at least 30 minutes of steady-state training showing images/s and step time consistent with finishing under 3 days, including checkpoint writes.
- Convergence evidence: a 1-epoch run on a stratified subset, or 2,000 steps, with loss and top-1 accuracy tracking the GPU baseline within a stated tolerance.
- Data-path evidence: OBS read amplification per epoch is near zero after caching, and a worker failure does not restart the epoch from scratch.
- Resilience evidence: checkpoint save/resume works from OBS, and the job survives a simulated preemption.
- Resource evidence: NPU memory headroom is reported, not assumed, so a slightly larger validation batch does not trigger OOM.
Only then would I commit the 3-day run. If any check fails, the decision is to fix that specific item, not to reduce batch size and hope.