MIG Layout Cannot Serve a Mixed Inference and Training Workload
You run a shared 8-GPU A100 80GB SXM node with NVLink. The inference team wants seven isolated 10 GB MIG slices per GPU for small models, and has already enabled MIG with the 1g.10gb profile on all eight GPUs. The training team now needs the same node for a 70B-parameter model using FSDP; its job fails with CUDA out-of-memory after seeing only 10 GB per GPU, and its NCCL initialization reports only one GPU per process. The platform team wants one configuration that serves both teams without buying new hardware. You must evaluate whether the 1g.10gb layout can satisfy training, explain the constraints MIG places on memory, NVLink peer access, and GPU discovery, and propose an operating model and validation steps.
1. Can the existing 1g.10gb MIG layout satisfy the 70B training job? Explain the memory, topology, and discovery constraints that drive your answer.
Read the worked answer
No. A 1g.10gb instance exposes roughly 10 GB and a fraction of the SMs, so a training rank cannot hold the layer working set even with FSDP sharding; optimizer states, activations, and communication buffers still require far more than 10 GB. MIG slices cannot be combined into one larger logical GPU: seven 10 GB instances do not form a 70 GB device. Each MIG instance also appears as a separate CUDA device, so a process sees one small GPU, not the whole A100. NVLink peer access must not be assumed to behave like full-GPU P2P across MIG instances; validate with cudaDeviceCanAccessPeer and an NCCL test. The training job therefore needs full-GPU mode, not a MIG profile.
2. What operating model would let inference use MIG and training use full GPUs on the same fleet with minimal disruption?
Read the worked answer
Partition by node rather than trying to satisfy both on one node at the same time. Dedicate some nodes to inference with MIG enabled and label them for MIG profiles; keep others in full-GPU mode for training. If hardware is fixed, use maintenance windows to toggle MIG: drain and cordon the node, stop workloads, disable MIG, reboot, verify full-GPU enumeration, then release to training; reverse for inference. Use Kubernetes labels, taints, and resource names so pods request nvidia.com/mig-1g.10gb or full nvidia.com/gpu and cannot land on the wrong node type. Validation should include nvidia-smi -L, nvidia-smi mig -lgip, MIG instance UUIDs, a MIG test pod, and an NCCL all-reduce on full-GPU nodes.
3. What validation would you require before declaring the mixed-mode operating model production-ready?
Read the worked answer
For MIG nodes, confirm the intended profile list with nvidia-smi mig -lgip, create instances with nvidia-smi mig -cgi, and verify each instance UUID is advertised by the device plugin. Run a small inference pod pinned to a MIG UUID and check nvidia-smi inside the container. For full-GPU nodes, confirm 8 devices, correct NVLink topology, and run NCCL all-reduce and nvbandwidth. Confirm scheduler behavior by submitting pods that request each resource type and checking they land only on compatible nodes. Add DCGM monitoring for XID, ECC, and thermal events, and document the drain, reboot, and rollback steps for MIG mode changes.