Containerized Training Fails Above 8 Nodes with NCCL Unhandled System Error

Rank container-to-host software mismatch first. The container carries CUDA 12.4 and MLNX_OFED 5.9, while the host has driver 535 and OFED 5.8. CUDA 12.4 containers generally expect a newer driver branch, and although NGC images may include compatibility libraries that let a single-node test run, multi-node RDMA registration exercises the kernel verbs path more heavily. An OFED 5.9 userspace in the container against 5.8 kernel modules can produce ibv_reg_mr failures when the ABI or provider behavior differs. Second, rank memory-registration limits: memlock is unlimited on the host, but the container may still inherit a lower locked-memory limit or cgroup restriction, and 16 nodes create more ranks and QPs than 8 nodes. Third, rank NCCL HCA selection and rail configuration; if NCCL picks a storage HCA or a mismatched rail, 16-node initialization can fail while smaller jobs fit within one leaf. Fourth, consider GPU Direct RDMA and nvidia-peermem mismatch. The fact that 8 nodes works but 16 fails points to scale-dependent resource exhaustion or cross-spine routing, but ibv_reg_mr specifically points to registration, not routing.

On a single node, compare host and container views: nvidia-smi driver version, nvcc --version or container CUDA version, ibv_devinfo, ldconfig -p | grep ibverbs, and ulimit -l inside the container. Verify that the container's libibverbs provider matches the host kernel modules and that /dev/infiniband includes the expected uverbs devices. Run a small ibv_reg_mr test or nccl-tests all-reduce inside the container and inspect dmesg for MR registration errors. On eight nodes, run with NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,NET to capture HCA selection, QP creation, and registration. Then run the 16-node test with NCCL_IB_HCA explicitly restricted to the compute HCAs, and repeat with a matching-version container such as CUDA 12.2/OFED 5.8. If the matching container passes 16 nodes, the root cause is version mismatch; if it still fails, check locked-memory and cgroup limits at scale. Also run the 16-node job on a single leaf first, then across the spine, to separate registration limits from fabric routing.

The safest short-term remediation is to rebuild or retag the container on a CUDA 12.2 and MLNX_OFED 5.8 base that matches the host driver and kernel verbs, then rerun the 16-node canary. This avoids host changes but may require rebuilding dependencies and validating the PyTorch stack. The alternative is to upgrade the host driver to a branch that supports CUDA 12.4 and align OFED to 5.9, which is cleaner long-term but introduces a cluster-wide maintenance window and regression risk across all jobs. A third option is to mount host OFED libraries into the container or use the NVIDIA Container Toolkit's driver capabilities so the container uses host verbs; this can work but must be tested because mixing userspace and kernel versions is exactly what caused the failure. Also set NCCL_IB_HCA explicitly, confirm memlock inside the container, and verify nvidia-peermem for GPU Direct RDMA. Whichever path you choose, require a 16-node NCCL test plus a representative training job before declaring success, and document the supported driver/CUDA/OFED matrix so the next image build does not reintroduce the mismatch.