PCIe Affinity and GPU Direct RDMA on a Mixed-Generation Fine-Tuning Cluster
A research lab runs parameter-efficient fine-tuning on 32 GPU servers, each with four A100 80 GB PCIe GPUs and one 200G RoCE NIC. Half are older PCIe Gen3 servers and half are Gen4. NCCL all-reduce across nodes reaches only 9 GB/s per GPU instead of the expected 20 GB/s. nvidia-smi topo -m shows the NIC attached to CPU socket 1 while two GPUs sit on socket 0, and NCCL logs report GPU Direct RDMA disabled. During collectives, host CPU utilization pins at 100%, and latency-sensitive inference jobs on the same hosts suffer. The lab plans to add 16 nodes with four GPUs each and 400G NICs, and wants one fabric and one operational model. Capital budget covers either more C-Series servers or a move to UCS X-Series with X440p PCIe nodes.
1. Diagnose the throughput gap. Which constraints are bandwidth, which are topology, and which are configuration?
Read the worked answer
Three separate problems are being conflated. First, bandwidth: a 200G NIC delivers about 25 GB/s in theory, shared by four GPUs. Even with perfect sharing that is about 6.25 GB/s per GPU, so a sustained 9 GB/s is already better than the naive budget and 20 GB/s per GPU is impossible with this adapter count. The server needs roughly 800G of NIC capacity to give each GPU 200G, or the workload must accept a lower per-GPU rate.
Second, topology: the NIC is on socket 1 and two GPUs are on socket 0, so those GPUs cross a UPI hop to reach the NIC. That adds latency and consumes inter-socket bandwidth. nvidia-smi topo -m will show SYS for those pairs rather than PIX or PXB. On Gen3 servers, a x16 link is about 15.75 GB/s theoretical and roughly 12.8 GB/s practical, shared between GPU and NIC traffic, which further caps throughput.
Third, configuration: GPU Direct RDMA is disabled, so NCCL bounces data through host memory. That explains 100% CPU and adds PCIe copies. Likely causes are ACS enabled on the PCIe root ports, IOMMU in a mode that blocks peer-to-peer, or a missing nvidia-peermem module. Fixing configuration alone will improve latency and CPU load but will not create the missing NIC bandwidth; fix bandwidth and affinity as well.
2. Recommend a placement and configuration standard for the existing 32 servers and for the 16 new nodes. What evidence would you require before accepting the design?
Read the worked answer
For the existing servers, place each NIC on the same PCIe switch or root complex as the GPUs it serves, and split NICs across sockets so each socket has local NIC capacity. On a four-GPU server, use two 200G NICs where possible, one per socket pair, rather than one NIC for all four. Enable GPU Direct RDMA by disabling ACS on the relevant root ports, configuring IOMMU for passthrough, loading nvidia-peermem, and confirming that nvidia-smi topo -m shows PIX or PXB between each GPU and its NIC. Set NCCL to use the correct interfaces and GID index and verify with all_reduce_perf, not just ping.
For the new nodes, choose a platform with PCIe Gen4 or Gen5 and enough lanes for four GPUs plus at least two 400G NICs. UCS X-Series with X440p PCIe nodes is defensible if the bill of materials confirms the GPU-to-NIC affinity and the power/cooling envelope; more C-Series is defensible if it preserves lane count and slot placement. Do not mix Gen3 and Gen4 nodes in the same NCCL communicator if avoidable; label them in the scheduler and test cross-generation rings separately.
Acceptance evidence: nvidia-smi topo -m for every node, NCCL debug logs showing GPU Direct RDMA enabled, a bus-bandwidth test on each node, and a multi-node all-reduce that meets an agreed GB/s per GPU target at the intended NIC count. Reject any design that only reports link speed.
3. The lab wants one fabric and one operational model across old and new servers. What should be standardized, and what should remain differentiated?
Read the worked answer
Standardize the policy, not the hardware. One RoCEv2 fabric can carry both generations if you standardize the DSCP/COS mapping, PFC and ECN policy, MTU, QoS queueing, and monitoring. Use the same leaf-spine design and the same automation templates, with variables for NIC speed, PCIe generation, and GPU count. That gives one operational model and one set of congestion counters.
Differentiate admission and placement. Old Gen3 servers should not be scheduled into jobs that require maximum inter-node bandwidth; expose a node label such as fabric_tier=standard versus fabric_tier=high and let the scheduler place jobs accordingly. New 400G nodes can be placed in a separate rail group or pod so a slow node does not set the collective rate for the whole job. NCCL uses the slowest link; mixing is therefore a scheduling policy problem, not only a fabric problem.
Also differentiate failure domains. Keep the old servers on leaf pairs that can be drained for maintenance without affecting the new pod. Monitor per-node achieved all-reduce bandwidth and feed it back to the scheduler so a degraded node is cordoned automatically. The target is one fabric with tiered service levels, not one homogeneous cluster.