NVIDIA AI Computing Platform and Architecture Fundamentals
The platform as a layered system
NVIDIA's AI platform is best understood as layers: silicon (GPU architectures such as Ampere, Hopper, and Blackwell), system-level interconnect (NVLink, NVSwitch, PCIe, and InfiniBand or Ethernet for scale-out), system software (drivers, CUDA, container runtime hooks), and higher-level SDKs. Questions often test whether you can place a component in the correct layer. A trap is confusing a hardware capability with a software feature: NVLink is a physical link, while NCCL is a library that exploits it.
Key architecture distinctions
Know the difference between a data-center GPU (high FP64 and matrix throughput, ECC memory, MIG partitioning) and a consumer GPU (fewer reliability features, no MIG). Understand that Tensor Cores accelerate matrix multiply-accumulate operations in reduced precision, while CUDA Cores handle general FP32/FP64 math. SMs (Streaming Multiprocessors) are the scheduling unit containing CUDA Cores, Tensor Cores, registers, and shared memory.
Multi-GPU topologies
Scale-up means fast intra-node links (NVLink/NVSwitch); scale-out means inter-node networking (InfiniBand, RoCE). A common trap: assuming PCIe bandwidth is sufficient for large-model training parallelism. It usually is not, which is why tensor parallelism stays inside a node.
Worked rule of thumb
If a model's per-GPU memory need exceeds one device, first check whether tensor parallelism fits within a single NVLink domain before considering pipeline or data parallelism across nodes. Rule of thumb: keep the most communication-heavy parallelism inside the fastest interconnect.
Precision and reliability
Be ready to distinguish FP32, TF32, FP16, BF16, FP8, and INT8, and which are used for training versus inference. MIG lets one GPU be partitioned into isolated instances; it is not the same as time-slicing, which shares the device without memory isolation.