Thermal-Induced XID 79 and NCCL Timeouts on an HGX H100 Cluster

First, stop the automatic restart loop for the affected allocation or drain gpu-17-04 so the same node is not repeatedly reinserted into a 512-GPU job. Preserve evidence before any reboot: dmesg -T around the XID 79 events, nvidia-smi -q -d TEMPERATURE,POWER,ECC,PAGE_RETIREMENT, BMC SEL logs, and the NCCL debug log from the failed rank. Record whether XID 79 is followed by a bus reset or a containment event; an XID 79 that clears after reset is still a reliability signal, not a benign log line. Check the node's PSU redundancy and power capping, because 92% PSU load plus 94 °C HBM suggests the node may be at a power/thermal edge. Compare the node's inlet and exhaust temperatures with its rack neighbors and inspect for missing blanking panels or reversed airflow. Do not immediately revert the facility setpoint: other nodes in the same rack are stable, so a global cooling rollback could hide a local hardware or airflow problem. Instead, isolate the node thermally and electrically, then reproduce with a controlled single-node burn-in at the current facility conditions.

Use a differential test rather than a single correlation. The facility hypothesis predicts multiple nodes in the same rack or row should show rising HBM temperatures and XIDs; currently only gpu-17-04 is named, so gather 24-hour temperature and XID data for all 32 nodes. If neighbors are below 85 °C HBM and XID-free, facility cooling is unlikely to be the primary cause. The node hypothesis predicts the fault follows the node: run a single-node GPU burn-in and an 8-GPU NCCL all-reduce on gpu-17-04 at the current inlet temperature. If it reproduces XID 79 or HBM throttling while an identical control node passes, treat it as node hardware until proven otherwise. Check page retirement and ECC counters, reseat or swap the GPU baseboard, and verify the heatsink and thermal interface if the platform allows. The workload hypothesis predicts the failure correlates with a specific collective pattern or memory access; inspect whether failures cluster on ranks using a particular NCCL algorithm or near checkpoint writes. Because XID 79 is a bus-fall-off event, workload alone is a weak explanation unless it is driving a power transient. A controlled burn-in plus a canary training job on the drained node gives the cleanest attribution.

Require a staged acceptance, not a single successful job. First, run at least 24 hours of single-node burn-in that includes sustained HBM traffic, power cycling, and repeated NCCL all-reduce across all eight GPUs; record HBM temperature, SM clock, power draw, and XID counters. Second, run an 8-node canary job that includes the same collective size and duration profile as the production job, then a 64-GPU job if the scheduler permits. Acceptance criteria should include no XID 79 or bus resets, no uncontained ECC errors, HBM temperature staying below the platform's throttle threshold with margin, and NCCL bandwidth within an agreed percentage of a known-good baseline. For monitoring, add per-GPU HBM temperature, XID count, PCIe replay, PSU redundancy state, and NCCL timeout alerts tagged by node. Set a lower temperature warning threshold than the throttle point so you get early warning after the chilled-water setpoint change. Finally, document whether the fix was hardware replacement, airflow correction, or power capping, because that determines whether the facility setpoint remains a cluster-wide risk.