Eight-GPU Node Fails Enumeration After Rack Integration
You are integrating four HGX H100 8-GPU servers into a single 42U rack. Each node has 8 SXM5 H100 GPUs, two Xeon Platinum CPUs, 8 NVMe drives, and four 3000 W PSUs; measured maximum node draw is 10.2 kW. The rack is fed by two 208 V 60 A PDUs and has a 20 kW rear-door heat exchanger. Inlet air is 22 C, but node 3 reports a 32 C intake because blanking panels were removed during cabling. After first power-on, node 3's BMC inventory shows 7 GPUs, nvidia-smi lists 7, and lspci shows the remaining GPU at x8 instead of x16. nvidia-smi nvlink -s reports an incomplete NVLink topology. The other three nodes enumerate all 8 GPUs. You must decide whether to accept node 3, how to fix the rack power and cooling design, and what acceptance tests to run before scheduling production training.
1. Triage the missing GPU and degraded PCIe link on node 3. What layers do you check, in what order, and when do you stop troubleshooting and replace hardware?
Read the worked answer
Start at the physical and firmware layers because one GPU is absent and the remaining link is x8, which is not a normal H100 SXM configuration. Reseat the GPU baseboard and its power connectors, inspect the PCIe riser and NVLink cartridge for bent pins or incomplete seating, and swap the suspect riser or cable with a known-good node. Check the BMC SEL and dmesg for PCIe correctable errors, link retraining, or power faults. In BIOS, confirm the slot is set to x16 and that bifurcation matches the riser. Run lspci -vv to confirm negotiated width and speed, and nvidia-smi nvlink -s to see which links are missing. If swapping known-good parts moves the fault, replace the FRU; if the same slot remains bad, inspect the baseboard. Do not accept the node: a missing GPU or degraded link changes NVLink topology and will silently reduce NCCL bandwidth.
2. Evaluate the rack power and cooling design. What is wrong, and what remediation would you require before production?
Read the worked answer
Power: four nodes at 10.2 kW = 40.8 kW. Two 208 V 60 A PDUs provide 2 x 12.48 kVA = 24.96 kVA; at 80% derating that is about 20 kW, less than half the load. Do not run this rack as designed. Spread nodes across additional racks or install higher-capacity three-phase circuits sized for 40.8 kW plus headroom. Cooling: a 20 kW rear-door unit cannot remove 40 kW; specify in-row or rack-level cooling rated above the actual load, restore blanking panels, and keep inlet temperature within the server's supported range. Measure per-node power with BMC telemetry and verify PDU phase balance. Acceptance should include a sustained power and thermal burn-in, not just idle boot.
3. Define the acceptance validation you would run on each node before releasing the rack to production.
Read the worked answer
Validate inventory and topology first: the BMC should show 8 GPUs; nvidia-smi -L should list 8; nvidia-smi topo -m should show the expected NVLink mesh with all 8 GPUs connected. Check PCIe negotiated width and speed for each GPU and NIC. Run nvbandwidth for GPU-to-GPU copy and a short NCCL all-reduce to confirm bandwidth. Run ECC and memory tests, then a 30-minute combined GPU and CPU burn-in while logging temperatures, clocks, and power. Confirm no XID errors in dmesg or DCGM. Only after all nodes pass should the rack be released; a node with a degraded link can pass a simple nvidia-smi check but fail at scale.