AMP accuracy regression after porting ConvNeXt to Ascend
A team ports an internal image classifier from A100 to Ascend 910B using torch_npu and CANN 7.0. The model is a ConvNeXt-B with LayerNorm, GELU, and dynamic input sizes from 512 to 1024 px. On GPU, validation top-1 is 78.4% in FP32 and 78.2% with AMP O1. On Ascend, FP32 reproduces 78.3%, but AMP O1 gives 74.9%. Logs show 14 ops fall back to CPU on some dynamic shapes, and ATC reports several subgraphs. The team proposes either switching everything to FP32 and accepting 2.3x lower throughput, or turning off dynamic shape entirely. The product needs 30 FPS at 640x640 and occasional 1024x1024 inspection. They ask for a decision.
1. Separate the accuracy drop into precision, CPU-fallback, and dynamic-shape causes. What experiments would you run to attribute the 3.5-point loss before changing the deployment configuration?
Read the worked answer
Start with the control: Ascend FP32 at 78.3% versus GPU FP32 at 78.4% shows the graph conversion and weights are essentially faithful; the loss is AMP-specific. A 3.5-point drop is too large to accept as normal FP16 rounding.
Run a fixed-shape 640x640 experiment first, so dynamic subgraphs are removed from the comparison. Then compare three variants against FP32: AMP with CPU fallback allowed, AMP with unsupported ops replaced so there is no fallback, and AMP with per-op precision overrides that keep LayerNorm, softmax, and final logits in FP32. Dump per-layer outputs for a fixed batch and compute cosine similarity and max absolute error against the FP32 run. Check ATC logs for subgraphs and CPU placement; CPU fallback can change reduction order and adds layout conversions, but it should not usually cost 3.5 points. If accuracy recovers only when normalization and logits are FP32, the cause is precision sensitivity. If it recovers only when fallback is removed, the cause is op/layout handling. Dynamic shape alone should change compilation and scheduling, not the mathematics, so test it last by comparing the same 640 image under static and dynamic OMs.
2. Recommend the final deployment configuration. Address precision policy, dynamic-shape handling, and the throughput target, and state why a blanket FP32 fallback is or is not justified.
Read the worked answer
A blanket FP32 deployment is not justified: it gives up 2.3x throughput to fix a problem that is probably confined to a few sensitive ops. Use a hybrid policy: keep convolutions and GELU in FP16/BF16, force LayerNorm, softmax, and the final classification layer to FP32, and remove CPU fallback by replacing unsupported ops with Ascend-supported equivalents or a custom TBE operator. Validate per-layer cosine similarity above 0.999 against FP32 and require top-1 at least 78.0% on the full validation set.
For shape handling, compile two static OM models for the common 640x640 path and the occasional 1024x1024 path, with batch sizes chosen for the 30 FPS requirement; add a small set of bucketed shapes if real inputs vary. This avoids per-image recompilation and CPU subgraphs while preserving coverage. Use AIPP for normalization and layout conversion, and batch frames on the 640 path. Turning off dynamic shape is acceptable only if the product can map every input size to a precompiled bucket; otherwise keep dynamic dimensions for the rare path. Measure end-to-end latency and accuracy on both OMs before release.