AI Infrastructure
What this domain actually tests
AI Infrastructure asks whether you can map a workload onto hardware: choosing GPUs, sizing memory, and understanding how accelerators, networks, storage, and power fit together in a data center.
Core concepts to own
GPUs are throughput machines: thousands of cores, many streaming multiprocessors, and tensor cores for matrix math. Learn the hierarchy of GPU memory bandwidth (HBM, measured in TB/s) versus interconnect bandwidth. Scale-up means fast intra-node links - NVLink and NVSwitch - connecting GPUs inside a DGX/HGX system. Scale-out means the cluster fabric: InfiniBand (NDR 400 Gb/s ≈ 50 GB/s per port) or Ethernet with RoCE/Spectrum-X. Know PCIe generations, GPU Direct Storage, and BlueField DPUs for offloading networking and security.
Understand MIG: it partitions a GPU into isolated slices with dedicated memory and SMs, useful for inference density and multi-tenant sharing, but not a way to exceed the GPU's aggregate bandwidth. Understand reference architectures, rack power density, and air versus liquid cooling as design constraints.
Traps
NVLink does not extend between nodes; questions that describe a 32-GPU NVLink domain are wrong. More GPUs rarely scale linearly because communication and synchronization dominate. HBM bandwidth (TB/s) and network bandwidth (GB/s) are different orders of magnitude - do not compare them loosely. MIG does not multiply total memory bandwidth.
Worked example
One H100 SXM offers about 3.35 TB/s HBM and 900 GB/s of NVLink per GPU, while a single NDR InfiniBand port gives ~50 GB/s. Intra-node bandwidth is therefore roughly 18× a single fabric port, which is why tensor-parallel training is kept inside a node where possible and pipeline or data parallelism crosses the fabric.