Troubleshooting Intermittent GPU Failures in a Shared Cluster
A shared research cluster has 32 GPU nodes. Users report that long training jobs fail intermittently, roughly once every 20 hours per job. Logs show a mix of out-of-memory errors, NCCL timeout errors, and occasional uncorrectable memory errors on one GPU. The cluster uses a scheduler with no health checks between jobs, and users sometimes run jobs that request more memory than the GPU has. One node has been rebooted twice for the same GPU error. The operations team wants to know whether to treat this as a user error problem, a hardware problem, or a scheduler problem, and how to reduce failure frequency.
1. How should the operations team separate the three failure signatures, and what does each imply?
Read the worked answer
The three signatures have different causes and should not be lumped together.
Out-of-memory errors: these are usually user or configuration problems. Jobs requesting more memory than available, or memory fragmentation from varying batch sizes, produce OOM. The fix is admission control and guidance: validate requested memory against the GPU capacity, document memory estimation, and consider per-job memory limits. OOM is not evidence of hardware failure.
NCCL timeouts: these indicate a communication stall, often caused by one rank being slow or dead. A GPU that throws an uncorrectable memory error can drop out of a collective, and the remaining ranks then time out. So NCCL timeouts may be a downstream symptom of the hardware fault, not an independent problem. The team should correlate timeout timestamps with hardware error events.
Uncorrectable memory errors: these are hardware faults. A node rebooted twice for the same GPU error should be drained and serviced, not returned to the pool. Continuing to schedule on it produces exactly the intermittent failures users see.
The operations team should treat this as all three, in priority order: hardware first, then scheduler health checks, then user guidance. The hardware fault is likely amplifying the other two by causing jobs to fail and restart, which increases memory pressure and collective timeouts.
2. What operational changes would reduce failure frequency, and how would you measure improvement?
Read the worked answer
Operational changes, in order of impact:
Drain and repair the faulty node. Quarantine any GPU that reports an uncorrectable memory error until it passes a diagnostic. Add a rule that repeated errors on the same GPU trigger automatic drain rather than reboot-and-return.
Add health checks between jobs. Before scheduling, run a short GPU health test and verify error counters are clean. This prevents known-bad hardware from receiving new work.
Add admission control for memory. Reject or warn on jobs whose requested memory exceeds the GPU capacity, and encourage checkpointing so restarts are cheap.
Improve failure attribution. Centralize GPU error counters, NCCL logs, and scheduler events with timestamps so a timeout can be traced to a hardware event on a specific node.
Encourage frequent checkpointing so that when a failure occurs, the job loses minutes rather than hours.
Measurement: track mean time between failures per job, the fraction of failures attributed to hardware versus user configuration versus communication, node-level error rates, and job restart cost. Compare a 4-week baseline against a 4-week period after the changes. Success looks like fewer hardware-attributed failures, a lower share of OOM failures after admission control, and shorter recovery time per failure.