Fabric and Scheduler Choices for a 512-GPU AI Factory Build
You are deploying a 64-node AI factory: 64 servers with 8 H100 GPUs each, for 512 GPUs, on an NDR 400 Gb/s InfiniBand compute fabric with a 25 GbE management network. The deadline is eight weeks. Forecast: 70% of GPU-hours will be large training jobs spanning 256–512 GPUs with all-reduce-heavy transformer workloads; 30% will be batch inference and small fine-tuning. Capital budget supports either a fully non-blocking 1:1 leaf-spine fabric or a 2:1 oversubscribed fabric; the difference is about $1.8M and two weeks of cabling and testing. Your team has deep Slurm experience and limited Kubernetes operations. The vendor proposes rail-optimized cabling and SHARP. Leadership wants the cheapest design that will not bottleneck the training roadmap.
1. Should you choose 1:1 or 2:1 oversubscription? Justify the decision against the workload forecast and the schedule.
Read the worked answer
Do not treat 2:1 as automatically acceptable just because 30% of GPU-hours are small jobs. The decisive question is where the 256–512-GPU training jobs run. If a 2:1 design places an entire 512-GPU job across oversubscribed spine uplinks, all-reduce traffic will hit incast and congestion control, and the job will lose far more than the $1.8M saved. A reasonable calculation: if 2:1 reduces effective training throughput by even 15% on 70% of GPU-hours, the lost GPU productivity over the cluster's service life usually exceeds the infrastructure delta. Therefore choose 1:1 for the spine, or negotiate a pod-based compromise only if the vendor can prove that 256-GPU jobs stay within non-blocking pods and that 512-GPU jobs are rare. Because the roadmap explicitly targets 512-GPU training, I would select 1:1 and protect the schedule by pre-terminating and pre-testing optics and cables. If the budget truly cannot absorb 1:1, reduce node count initially rather than building an oversubscribed fabric that cannot be upgraded without recabling. The two-week testing cost is small compared with repeated congestion debugging during peak training.
2. How would you design the compute fabric and cabling, including rail optimization and SHARP, given the vendor proposal?
Read the worked answer
Design for rail-aligned GPU-to-leaf mapping: GPU 0 on every node connects to leaf 0, GPU 1 to leaf 1, and so on, so that a multi-node all-reduce can use eight parallel rails without traversing a spine for every packet. With 64 nodes and eight rails, each leaf group serves one GPU index per node; size uplinks so that each rail has enough spine capacity for the largest expected collective. Use a two-tier leaf-spine with 1:1 subscription and separate management and storage traffic on the 25 GbE network. Validate that the vendor's cabling plan preserves rail alignment at every patch panel; a single swapped rail degrades NCCL performance in ways that look like application jitter. Enable SHARP only after baseline NCCL tests pass without it, then compare all-reduce bus bandwidth and CPU overhead. SHARP can reduce switch traffic for large collectives, but it adds operational complexity and firmware dependencies. Acceptance should include per-rail link width and BER checks, NCCL tests at 8, 64, 256, and 512 GPUs, and a failover test with one leaf uplink disabled.
3. The team knows Slurm well but has limited Kubernetes operations. What scheduler and provisioning approach do you recommend for the eight-week rollout?
Read the worked answer
Choose Slurm as the primary scheduler for the training fabric and use NVIDIA Base Command Manager or an equivalent provisioning layer for image lifecycle and node health. The team's operational depth matters more than theoretical scheduler features: an eight-week deadline leaves little room to build Kubernetes expertise, CSI drivers, GPU operator tuning, and multi-node job debugging from scratch. Slurm with Pyxis/Enroot or Singularity covers containerized training, GPU affinity, and gang scheduling for 512-GPU jobs. If inference requires Kubernetes later, dedicate a separate partition or separate pool rather than forcing both workloads through one orchestration stack on day one. Provision nodes with a known-good image, pinned driver and CUDA versions, and automated burn-in that runs before a node joins the partition. Define a canary pool of 8 nodes for scheduler and fabric integration, then expand to 64. Your acceptance gate should be a representative 256-GPU training job running for at least several hours with stable NCCL bandwidth, not merely a scheduler ping test. This approach protects the deadline while leaving a migration path for inference services.