I85PREP
⌘ K
NVIDIA NCP-AII study materials
NVIDIA NCP-AII study resources

NVIDIA NCP-AII: Cluster Orchestration and Workload Management

NVIDIA NCP-AII · Work through your exam, one topic at a time.

All guide sections

Cluster Orchestration and Workload Management

What this domain tests

Scheduling GPUs without oversubscribing them: Slurm GRES and MIG resource types, Kubernetes device plugin and GPU Operator behavior, topology-aware placement, and how slicing modes change what a pod or job can request.

Core rules

  • Slurm. GPUs are consumed through GRES. SelectType=select/cons_tres plus NVML autodetection (GresTypes=gpu with Autodetect=nvml) lets Slurm discover devices; jobs then request --gpus=8 or --gpus-per-task. MIG slices are declared as distinct GRES types in gres.conf so a job requests a specific profile rather than a whole GPU.
  • Kubernetes. The NVIDIA GPU Operator bundles the driver container, device plugin, GPU Feature Discovery, DCGM exporter, and MIG manager. Pods request nvidia.com/gpu, and GFD labels such as nvidia.com/gpu.product let you pin workloads to a GPU model.
  • MIG changes the resource name. With MIG enabled, pods request nvidia.com/mig-1g.5gb style resources; requesting nvidia.com/gpu will leave them Pending.
  • Time-slicing multiplies allocatable GPUs but provides no memory isolation and no fault isolation; MPS shares a GPU with better concurrency but is still a single failure domain.

Worked example

A pod stays Pending with "0/8 nodes available: 8 Insufficient nvidia.com/gpu", yet nvidia-smi works on every node. Reasoning: the request name does not match the advertised resource. Check kubectl describe node | grep nvidia.com - if the node advertises only nvidia.com/mig-1g.5gb, the workload's nvidia.com/gpu request can never be satisfied. Either change the pod spec to the MIG resource name or reconfigure the MIG manager profile.

Traps that cost marks

Assuming a scheduled GPU job means the container can see the device - verify from inside; requesting whole GPUs on a MIG-partitioned node; expecting memory isolation from time-slicing.