I85PREP
⌘ K
NVIDIA NCP-AIO study materials
NVIDIA NCP-AIO study resources

NVIDIA NCP-AIO practice questions

One question at a time. Build your NVIDIA NCP-AIO understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1AI Infrastructure and Data Center Operations

In a DGX SuperPOD deployment, which fabric is primarily responsible for carrying GPU-to-GPU collective traffic such as all-reduce during large-model training?

  1. The BMC baseboard management controller link
  2. The InfiniBand or RoCE east-west compute fabric Correct answer.
  3. The out-of-band 1GbE management network
  4. The north-south storage front-end network

Explanation

Training collectives exchange gradients and activations directly between GPUs, so they ride the low-latency, high-bandwidth east-west fabric (InfiniBand or RoCEv2). Management, BMC, and storage front-end networks serve control, telemetry, and data-loading roles and are not sized for collective traffic.

Question 2AI Infrastructure and Data Center Operations

A data center operator needs to confirm that a GPU node's power draw stays within the rack's per-circuit limit before enabling full training load. Which telemetry source gives the most direct measurement of node-level power consumption?

  1. InfiniBand port error counters
  2. NVIDIA driver version strings
  3. GPU clock frequency counters
  4. Baseboard management controller power sensors Correct answer.

Explanation

The BMC exposes platform power sensors (PSU input and total board power) that reflect actual node draw. GPU clocks, fabric counters, and driver versions describe performance or configuration, not wall power, so they cannot validate circuit loading.

Question 3AI Infrastructure and Data Center Operations

Which rack-level design practice best supports the high power density of modern GPU nodes such as 8-GPU HGX systems?

  1. Relying on room-level CRAC units sized for average rather than peak load
  2. Concentrating GPU nodes in liquid-cooled or high-density racks with adequate CDU capacity Correct answer.
  3. Placing GPU nodes adjacent to networking gear to shorten cable runs
  4. Distributing nodes across many low-density racks to reduce per-rack heat

Explanation

High-density GPU nodes can draw far more per rack unit than traditional servers, so operators use liquid cooling or purpose-built high-density racks with coolant distribution units sized for peak load. Spreading nodes out wastes fabric reach, and room-level cooling sized for average load will throttle under sustained training.

Question 4AI Infrastructure and Data Center Operations

An operator wants to isolate GPU training traffic from storage traffic on the same physical NIC using RoCEv2. Which mechanism is designed for this purpose?

  1. VLAN tagging with priority flow control on separate VLANs Correct answer.
  2. Assigning identical IP addresses to both traffic classes
  3. Using jumbo frames on a single untagged VLAN
  4. Disabling ECN on the storage interfaces

Explanation

Separate VLANs let operators apply distinct QoS, PFC, and ECN policies per traffic class so lossless GPU traffic is not disrupted by storage bursts. Jumbo frames, disabled ECN, or duplicate addressing do not create isolation and can worsen congestion or break routing.

Question 5AI Infrastructure and Data Center Operations

A cluster administrator wants a single pane to monitor GPU temperature, ECC errors, and NVLink status across all nodes. Which NVIDIA tool is purpose-built for this fleet-level health view?

  1. NVIDIA DCGM Correct answer.
  2. NVIDIA Container Toolkit
  3. nvidia-smi run interactively on each node
  4. CUDA Toolkit profiler

Explanation

DCGM is the data center GPU manager designed to collect health, telemetry, and diagnostic metrics across a fleet and expose them to monitoring stacks. Running nvidia-smi per node does not scale, while the container toolkit and CUDA profiler serve different purposes.

Question 6AI Infrastructure and Data Center Operations

Which statement accurately describes the role of a rail-optimized network topology in a large GPU cluster?

  1. Rails are used only for management traffic to isolate BMC access
  2. Rail optimization removes the need for any spine layer
  3. Each GPU connects to a leaf switch on a dedicated rail, so same-index GPUs communicate over one hop Correct answer.
  4. All GPUs on a node share a single uplink to reduce switch port count

Explanation

Rail-optimized designs map each GPU position to its own leaf, so the same GPU index across nodes reaches its peers in a single hop, minimizing collective latency. Sharing one uplink or reserving rails for management defeats the purpose, and spines are still needed for cross-rail traffic.

Question 7AI Infrastructure and Data Center Operations

During a large all-reduce, operators observe that RoCEv2 links are dropping packets and throughput collapses. Which combination of settings is most appropriate to make the fabric lossless?

  1. Disable PFC and rely on TCP retransmission
  2. Increase MTU to 9216 and disable flow control
  3. Enable PFC on all priorities and disable ECN marking
  4. Enable PFC on a dedicated priority and configure ECN marking with DCQCN Correct answer.

Explanation

Lossless RoCEv2 uses priority flow control on a dedicated traffic class plus ECN marking so DCQCN can throttle senders before buffers overflow. Enabling PFC everywhere risks head-of-line blocking and deadlocks, while disabling flow control or ECN allows drops that devastate collective performance.

Question 8AI Infrastructure and Data Center Operations

An administrator is provisioning a new GPU node and needs to verify that the correct NVIDIA driver and CUDA versions are installed before joining it to the training cluster. Which command provides this verification?

  1. ipmitool sensor list
  2. dcgmi discovery -l
  3. ibstat and ibv_devinfo
  4. nvidia-smi and nvcc --version Correct answer.

Explanation

nvidia-smi reports the loaded driver and CUDA runtime compatibility, while nvcc --version reports the toolkit compiler version. The other commands report hardware sensors, InfiniBand state, or GPU inventory, none of which confirm driver and toolkit versions.

Question 9AI Infrastructure and Data Center Operations

A GPU node repeatedly reboots under sustained training load but passes idle diagnostics. Which operational step best isolates whether the cause is power delivery or thermal?

  1. Replace the InfiniBand cable to the leaf switch
  2. Reinstall the NVIDIA driver on the node
  3. Correlate BMC power and inlet/GPU temperature logs around the reboot events Correct answer.
  4. Increase the batch size to reproduce faster

Explanation

Reboots under load often stem from PSU overcurrent, voltage droop, or thermal shutdown, and BMC logs timestamp power and temperature at the moment of failure. Reinstalling drivers, swapping cables, or increasing load does not distinguish power from thermal causes and may worsen the fault.

Question 10AI Infrastructure and Data Center Operations

Which practice best protects training job continuity when a single GPU in a node develops uncorrectable ECC errors?

  1. Use DCGM health checks and job checkpointing so the node can be drained and the job resumed Correct answer.
  2. Ignore the errors because ECC corrects all faults
  3. Move the job to a CPU-only node permanently
  4. Disable ECC on all GPUs to prevent job failures

Explanation

DCGM can detect uncorrectable ECC errors and mark the GPU unhealthy, while periodic checkpointing lets the scheduler drain the node and resume work elsewhere. Disabling ECC or ignoring errors risks silent data corruption, and abandoning GPUs sacrifices the platform.

Question 11AI Infrastructure and Data Center Operations

An operator must ensure that containers in a Kubernetes cluster can access GPUs. Which component is required on each GPU node to expose the devices to the container runtime?

  1. A CPU-only device plugin
  2. The CUDA profiler daemon
  3. The NVIDIA GPU Operator with the device plugin Correct answer.
  4. The InfiniBand subnet manager

Explanation

The GPU Operator installs drivers, the container toolkit, and the NVIDIA device plugin that advertises GPUs as schedulable resources. CPU plugins, profilers, and subnet managers do not expose GPU devices to containers.

Question 12AI Infrastructure and Data Center Operations

A cluster uses a fat-tree InfiniBand fabric. Users report that cross-rack collective performance is far below in-rack performance. Which investigation is most relevant?

  1. Check the container image registry latency
  2. Check the CUDA toolkit patch level
  3. Check the BMC firmware version on compute nodes
  4. Check spine-layer oversubscription and adaptive routing balance Correct answer.

Explanation

Cross-rack traffic traverses spine switches, so oversubscription ratios and adaptive routing balance determine achievable bandwidth. BMC firmware, CUDA patch levels, and registry latency do not explain a rack-local versus cross-rack gap.

Question 13AI Infrastructure and Data Center Operations

Which metric is the most direct indicator of congestion on a RoCEv2 compute fabric?

  1. ECN marked packet counters and PFC pause frame counts Correct answer.
  2. Filesystem inode usage
  3. GPU SM utilization percentage
  4. Container restart counts

Explanation

ECN marking and PFC pause frames are generated when queues build toward congestion, so their counters directly reveal fabric pressure. GPU utilization, container restarts, and inode usage describe compute, application, and storage state rather than network congestion.

Question 14AI Infrastructure and Data Center Operations

A data center team is planning cooling for a row of 8-GPU nodes drawing roughly 10 kW each. Which approach is most appropriate for sustaining peak training load?

  1. Direct-to-chip liquid cooling with rear-door heat exchangers or CDUs sized for peak Correct answer.
  2. Traditional room CRAC units sized for idle power draw
  3. Reducing GPU clocks permanently to fit air cooling
  4. Open-air racks with no containment

Explanation

Sustained high-density GPU load requires liquid cooling or rear-door heat exchange with CDUs sized for peak heat rejection. Room cooling sized for idle, open racks, or permanent downclocking either fails to remove heat or sacrifices performance.

Question 15AI Infrastructure and Data Center Operations

An operator needs to confirm that a newly cabled InfiniBand port is up and negotiating at the expected link width and speed. Which command is most appropriate?

  1. nvidia-smi -q -d TEMPERATURE
  2. dcgmi health -c
  3. ibstat Correct answer.
  4. ipmitool chassis status

Explanation

ibstat reports InfiniBand port state, active link width, and active speed, which is exactly what cabling validation requires. The other commands report GPU temperature, GPU health, or chassis power state.

Question 16AI Infrastructure and Data Center Operations

Which practice best supports reproducible GPU node provisioning across a large cluster?

  1. Using different driver versions per rack to spread risk
  2. Golden images or declarative configuration with version-pinned drivers and firmware Correct answer.
  3. Manually installing packages on each node as needed
  4. Allowing each node to auto-update firmware from the vendor at boot

Explanation

Reproducibility requires a known-good image or declarative configuration with pinned driver and firmware versions so every node matches. Manual installs, uncontrolled auto-updates, and deliberate version drift create inconsistency that complicates debugging and scheduling.

Question 17AI Infrastructure and Data Center Operations

A scheduler must place a distributed training job so that its GPUs communicate over a single rail where possible. Which capability enables this?

  1. Random GPU assignment across nodes
  2. Pinning all jobs to a single node
  3. Topology-aware scheduling using fabric and rail information Correct answer.
  4. Disabling the device plugin

Explanation

Topology-aware schedulers use node and fabric topology metadata to co-locate communicating GPUs on the same rail or leaf, reducing hops. Random placement, single-node pinning, or disabling the device plugin either ignores topology or removes GPU scheduling entirely.

Question 18AI Infrastructure and Data Center Operations

An operator observes that GPU memory errors are rising on several nodes but the GPUs still pass basic tests. Which action best follows data center operations best practice?

  1. Disable error reporting to reduce log noise
  2. Increase training load to stress the GPUs further
  3. Run DCGM diagnostics at increasing levels and schedule replacement for failing GPUs Correct answer.
  4. Ignore the errors until the GPUs fail outright

Explanation

Escalating DCGM diagnostic levels can distinguish marginal hardware from transient issues and support a replacement decision before failure. Adding load, suppressing logs, or waiting for hard failure risks job corruption and unplanned outages.

Question 19AI Infrastructure and Data Center Operations

A cluster's RoCEv2 fabric shows PFC pause frames storming across many priorities, stalling unrelated traffic. Which remediation is most appropriate?

  1. Increase PFC watchdog timers only
  2. Restrict PFC to the dedicated lossless priority and verify QoS mapping end to end Correct answer.
  3. Enable PFC on all remaining priorities
  4. Remove ECN marking to reduce pause frames

Explanation

Pause storms usually result from PFC being enabled on priorities that carry non-lossless traffic or from inconsistent QoS mapping. Restricting PFC to the designated lossless class and validating DSCP-to-priority mapping end to end prevents pause propagation. Timer changes, broader PFC, or removing ECN do not fix the mapping root cause.

Question 20AI Infrastructure and Data Center Operations

An operator needs to plan maintenance so that draining a single GPU node does not fail a large synchronous training job. Which design element is most important?

  1. Pinning the job to one node to avoid drain effects
  2. Frequent checkpointing combined with elastic or restartable job scheduling Correct answer.
  3. Disabling health checks during maintenance windows
  4. Running the job without checkpoints to maximize speed

Explanation

Checkpointing preserves progress, and elastic or restartable scheduling lets the job continue or resume after a node is drained. Running without checkpoints, pinning to one node, or disabling health checks increases the risk and impact of a drain.

Question 21AI Infrastructure and Data Center Operations

A data center team wants to detect a failing power supply before it causes a node outage. Which monitoring approach is most effective?

  1. Track InfiniBand symbol error rates
  2. Poll GPU utilization every hour
  3. Monitor container image pull times
  4. Trend BMC PSU input power and status flags and alert on anomalies Correct answer.

Explanation

PSU status flags and input power trends reveal degradation such as voltage anomalies or a supply dropping offline before redundancy is lost. GPU utilization, image pull times, and fabric error rates do not reflect PSU health.

Question 22AI Infrastructure and Data Center Operations

Which consideration most affects the choice between InfiniBand and RoCEv2 for a new large-scale GPU fabric?

  1. Operational maturity of lossless configuration and existing team expertise Correct answer.
  2. The number of CPU cores per node
  3. The size of the container registry
  4. The color of the fiber patch cables

Explanation

Both fabrics can carry GPU traffic, but they differ in configuration complexity, congestion control maturity, and the skills required to operate them losslessly. Cable color, CPU core count, and registry size do not determine fabric technology choice.

Question 23AI Infrastructure and Data Center Operations

Which statement correctly describes the purpose of a subnet manager in an InfiniBand fabric?

  1. It schedules Kubernetes pods onto GPU nodes
  2. It monitors GPU temperature across nodes
  3. It configures routing and link parameters so all ports can communicate Correct answer.
  4. It manages container image layers

Explanation

The subnet manager discovers the topology and programs forwarding tables and link parameters so InfiniBand ports can communicate. GPU monitoring, pod scheduling, and image management are handled by other components.

Question 24AI Infrastructure and Data Center Operations

Before returning a repaired GPU node to production, which validation sequence best confirms it is ready for training?

  1. Run only a ping test to the management interface
  2. Run driver and firmware checks, DCGM diagnostics, then a fabric link test Correct answer.
  3. Boot the node and immediately start the largest training job
  4. Skip validation to save maintenance time

Explanation

A layered validation confirms software versions, GPU health, and fabric connectivity before the node accepts production work. A ping test, immediate full load, or skipping validation can return a faulty node to service and cause job failures.

Question 25AI Infrastructure and Data Center Operations

An operator is asked to reduce the risk of thermal throttling during peak training hours. Which combination is most appropriate?

  1. Raise the data hall setpoint to save energy
  2. Block rear airflow to reduce dust ingress
  3. Disable temperature alerts to avoid noise
  4. Verify cooling capacity and airflow containment, then monitor GPU temperature and clock throttling metrics Correct answer.

Explanation

Throttling prevention requires adequate cooling capacity, proper hot/cold aisle containment, and telemetry that shows temperature and clock throttling in real time. Raising setpoints, blocking airflow, or silencing alerts worsens thermal conditions or hides them.

Question 26AI Infrastructure and Data Center Operations

In a data center hosting dense multi-GPU training nodes, which metric most directly reflects the electrical efficiency of power delivery from the facility input to the compute load?

  1. Rack-level power draw compared with the rack power budget
  2. Server inlet temperature compared with ASHRAE recommended limits
  3. Power usage effectiveness measured at the building level
  4. Power conversion efficiency of the UPS and PDU chain at the operating load Correct answer.

Explanation

Power conversion efficiency describes how much of the input energy reaches the IT load after UPS and PDU conversion losses, which is the direct measure of delivery efficiency. Facility-wide PUE blends cooling and other overheads, so it cannot isolate power delivery performance.

Question 27AI Infrastructure and Data Center Operations

A data center operator is preparing a hall for high-density GPU racks and wants to prevent hot exhaust air from recirculating into equipment intakes. Which containment strategy addresses this goal?

  1. Raising the chilled water supply temperature to reduce chiller energy
  2. Hot-aisle containment with blanking panels installed in unused rack units Correct answer.
  3. Increasing the CRAC fan speed to raise the overall airflow volume
  4. Deploying additional humidity sensors throughout the cold aisles

Explanation

Hot-aisle containment physically separates hot exhaust from cold supply air, and blanking panels stop bypass airflow through empty rack space. Together they reduce recirculation into equipment intakes, which is the stated objective.

Question 28AI Infrastructure and Data Center Operations

An operator is drafting a change-management procedure for firmware updates on GPU nodes in a production cluster. Which control most directly limits the blast radius of a faulty firmware release?

  1. Canary-updating a small node pool and validating workloads before fleet rollout Correct answer.
  2. Requiring the update to be applied manually by each node owner
  3. Applying the update to the entire fleet during a single maintenance window
  4. Scheduling the update during peak hours to shorten the total outage

Explanation

A canary rollout exposes only a small fraction of the fleet to the new firmware and allows validation with representative workloads before wider deployment. This bounds the number of nodes affected if the release proves faulty.

Question 29AI Infrastructure and Data Center Operations

A site reliability engineer must define service level objectives for a GPU training platform. Which measurement best represents availability from the user's perspective?

  1. Percentage of GPU nodes reporting healthy BMC status
  2. Count of firmware versions currently deployed in the fleet
  3. Average GPU utilization across the cluster over a month
  4. Proportion of submitted jobs that start and complete within their declared deadline Correct answer.

Explanation

A user-centric availability objective measures whether submitted work actually starts and finishes within its promised deadline. That captures scheduler, fabric, and node health together as the user experiences them.

Question 30AI Infrastructure and Data Center Operations

An operator notices that job start latency spikes whenever a large batch of short jobs is submitted to the GPU cluster scheduler. Which action best reduces this queueing delay without overcommitting GPUs?

  1. Reducing the number of scheduler replicas to lower overhead
  2. Disabling preemption so running jobs are never interrupted
  3. Enabling gang scheduling with backfill for short jobs that fit available gaps Correct answer.
  4. Increasing the maximum job runtime limit for all users

Explanation

Backfill lets short jobs use reserved but currently idle capacity as long as they finish before the reservation begins, which cuts head-of-line blocking. Gang scheduling keeps multi-GPU jobs atomic so partial allocations do not strand resources.

Question 31AI Infrastructure and Data Center Operations

A monitoring system must alert when a GPU node's memory bandwidth degrades relative to its own historical baseline. Which approach is most appropriate?

  1. Weekly manual review of bandwidth logs by an operator
  2. Static thresholds set from the vendor's theoretical peak bandwidth
  3. Alerting only when the node's GPUs exceed 95 percent utilization
  4. Per-node rolling baselines with anomaly detection on bandwidth counters Correct answer.

Explanation

Per-node rolling baselines account for workload and configuration differences and detect drift from each node's own normal behavior. Anomaly detection on bandwidth counters catches gradual degradation that fixed thresholds would miss.

Question 32AI Infrastructure and Data Center Operations

An operator must roll out a driver upgrade across a GPU cluster while minimizing disruption to long-running training jobs. Which sequence is most appropriate?

  1. Ask users to restart jobs at their convenience while nodes upgrade in place
  2. Upgrade only the head node and rely on version skew tolerance
  3. Upgrade all nodes simultaneously during a declared maintenance window
  4. Cordon and drain nodes in waves, upgrade drained nodes, validate, then uncordon Correct answer.

Explanation

Draining in waves lets running jobs migrate or checkpoint before the node changes, and validation before uncordoning prevents bad nodes from rejoining. Wave-based rollout also limits the fraction of capacity offline at once.

Question 33AI Infrastructure and Data Center Operations

A team needs to trace which tenant consumed GPU hours during a billing period and reconcile that against node-level telemetry. Which data source is most suitable?

  1. Fabric switch port counters
  2. Scheduler job records joined with GPU allocation telemetry Correct answer.
  3. BMC event logs on each GPU node
  4. Data center DCIM asset inventory

Explanation

Scheduler records identify which job and tenant held which GPUs for how long, and allocation telemetry confirms actual usage. Joining the two produces auditable per-tenant accounting.

Question 34AI Infrastructure and Data Center Operations

During an incident, an operator finds that a single leaf switch in the compute fabric is dropping counters while its uplinks remain healthy. Which troubleshooting step should come first?

  1. Raise the MTU on all fabric links
  2. Replace the switch immediately to restore redundancy
  3. Check the transceivers and cabling on the affected downlinks Correct answer.
  4. Reboot the entire fabric to clear transient errors

Explanation

Localized drops on one leaf with healthy uplinks point to the downlink path, so inspecting optics and cabling on those ports is the least invasive first step. Confirming physical-layer health before replacing hardware avoids unnecessary swaps.

Question 35AI Infrastructure and Data Center Operations

An operator wants to ensure that a failed GPU node is automatically removed from the scheduling pool before users submit new jobs to it. Which mechanism best achieves this?

  1. A manual weekly audit of node status reports
  2. Periodic emails to users listing known-bad nodes
  3. Increasing the scheduler's retry count for failed pods
  4. Node health checks that taint or cordon the node when GPU errors are detected Correct answer.

Explanation

Automated health checks that taint or cordon a failing node remove it from eligibility as soon as errors are detected, before new work lands on it. This shifts detection from human processes to continuous automation.

Question 36AI Infrastructure and Data Center Operations

A facility team is evaluating liquid cooling for a new high-density GPU hall. Which statement correctly characterizes direct-to-chip liquid cooling?

  1. It removes heat from cold plates on high-power components via a coolant loop Correct answer.
  2. It requires the entire server to be submerged in a dielectric tank
  3. It eliminates the need for any facility chilled water loop
  4. It only cools the storage shelves and not the compute nodes

Explanation

Direct-to-chip cooling attaches cold plates to high-power components such as GPUs and CPUs and carries heat away through a coolant loop to a facility heat rejection system. It is not immersion and does not remove the need for facility heat rejection.

Question 37AI Infrastructure and Data Center Operations

An operator is designing alerting for a GPU cluster and wants to avoid alert fatigue while still catching real failures. Which design is most appropriate?

  1. Tier alerts by severity with deduplication and maintenance-window suppression Correct answer.
  2. Disable alerting for nodes in production to avoid noise
  3. Send all alerts to a shared mailbox checked once per day
  4. Alert on every counter threshold breach with no suppression

Explanation

Severity tiers separate page-worthy conditions from informational ones, while deduplication and maintenance suppression prevent repeated or expected alerts. This preserves signal for genuine failures without flooding responders.

Question 38AI Infrastructure and Data Center Operations

A capacity planner must forecast GPU demand growth for the coming year. Which input combination yields the most defensible forecast?

  1. Facility power capacity alone
  2. Historical job submission trends, queue wait times, and planned project pipelines Correct answer.
  3. Current GPU utilization alone
  4. Vendor roadmap announcements alone

Explanation

Combining demand-side history such as submissions and queue waits with known upcoming project pipelines captures both observed growth and committed future load. Utilization alone or single external inputs omit key drivers.

Question 39AI Infrastructure and Data Center Operations

An operator must decide whether to place a latency-sensitive inference service and a batch training workload on the same GPU pool. Which analysis is most relevant?

  1. Checking whether both workloads use the same container base image
  2. Confirming that both workloads were submitted by the same team
  3. Evaluating contention for GPU memory, SM occupancy, and fabric bandwidth between the two workloads Correct answer.
  4. Comparing the color of the racks hosting each workload

Explanation

Co-location risk depends on whether the workloads compete for the same scarce resources: GPU memory, compute occupancy, and interconnect bandwidth. Quantifying that contention determines whether isolation or dedicated pools are needed.

Question 40AI Infrastructure and Data Center Operations

A cluster operator observes that after a firmware update, a subset of nodes intermittently fail GPU enumeration at boot but succeed on a second reboot. Which root-cause investigation is most appropriate?

  1. Rolling back the entire cluster to the prior driver without collecting data
  2. Increasing the number of retries in the scheduler
  3. Correlating the affected node serials with the firmware version and checking initialization timing logs Correct answer.
  4. Assuming the GPUs are failing and ordering replacements

Explanation

Intermittent enumeration that resolves on retry suggests a timing or initialization race introduced by the firmware change, so correlating affected serials with firmware version and boot timing logs isolates the cause. Collecting evidence before rollback preserves the ability to fix the issue properly.

Question 41AI Factory Deployment and Management

In an NVIDIA AI factory reference architecture, which fabric is primarily responsible for carrying collective operations such as all-reduce between GPU nodes during distributed training?

  1. The BMC and IPMI control plane
  2. The GPU-to-GPU scale-out compute fabric Correct answer.
  3. The storage-facing data network
  4. The front-end Ethernet management network

Explanation

Distributed training collectives such as all-reduce are latency- and bandwidth-sensitive and therefore run over the dedicated GPU scale-out fabric, typically InfiniBand or high-speed RoCE. Management, BMC, and storage networks carry control traffic and dataset I/O rather than the collective synchronization traffic itself.

Question 42AI Factory Deployment and Management

A team is deploying a large GPU cluster and wants to validate that the compute fabric delivers expected bandwidth before handing the cluster to users. Which practice best fits an AI factory bring-up workflow?

  1. Test each node's GPU memory bandwidth with a single-node microbenchmark
  2. Run a fabric-level bandwidth and collective benchmark across nodes and compare against expected line rate Correct answer.
  3. Verify only that the orchestrator can schedule a pod onto a GPU node
  4. Rely on link-up status lights on the switches alone

Explanation

Acceptance testing of an AI factory compute fabric requires running inter-node bandwidth and collective benchmarks and comparing measured results to the expected line rate for the design. Link lights, single-node memory tests, and scheduling checks confirm presence of components but do not prove the fabric performs as designed.

Question 43AI Factory Deployment and Management

Which statement best describes the role of a cluster manager such as Slurm or Kubernetes in an NVIDIA AI factory?

  1. It replaces the need for a GPU driver on each node
  2. It provides the physical layer encoding for the compute fabric
  3. It schedules workloads onto nodes and manages resource allocation and lifecycle Correct answer.
  4. It sets the GPU clock frequency during training jobs

Explanation

Cluster managers allocate compute resources and place workloads onto nodes, handling queueing, scheduling, and lifecycle of jobs or pods. They sit above the drivers and fabric and do not perform physical-layer encoding or clock control themselves.

Question 44AI Factory Deployment and Management

An operator needs to update GPU drivers across a large cluster with minimal disruption to running training jobs. Which approach aligns with AI factory management best practice?

  1. Disable the scheduler and reboot the entire cluster at once
  2. Update all nodes simultaneously during peak training hours
  3. Skip validation after updates to reduce downtime
  4. Drain nodes in batches, update, validate, and return them to service Correct answer.

Explanation

Rolling updates that drain a subset of nodes, apply the change, validate, and then return them to service limit blast radius and preserve availability for other jobs. Simultaneous cluster-wide changes or skipping validation increase the risk of prolonged outages and undetected regressions.

Question 45AI Factory Deployment and Management

Which component is responsible for out-of-band management of a GPU server, such as power control and console access, independent of the operating system?

  1. The job scheduler
  2. The GPU driver stack
  3. The container runtime
  4. The baseboard management controller Correct answer.

Explanation

The baseboard management controller provides out-of-band access for power control, console, and hardware telemetry even when the OS is down. Drivers, container runtimes, and schedulers all operate within or above the host operating system and cannot function when it is unavailable.

Question 46AI Factory Deployment and Management

A data center team is planning power and cooling for a dense GPU rack. Which consideration is most characteristic of AI factory deployment planning?

  1. Planning for high per-rack power density and associated liquid or high-capacity cooling Correct answer.
  2. Assuming GPU racks draw the same power as general-purpose compute racks
  3. Sizing cooling based only on switch power draw
  4. Ignoring airflow because GPUs are air-cooled by definition

Explanation

GPU-dense racks draw far more power per rack than general-purpose compute, so deployment planning must account for high power density and often liquid cooling or high-capacity air cooling. Assuming parity with standard racks or ignoring airflow leads to under-provisioned facilities.

Question 47AI Factory Deployment and Management

What is the primary purpose of NVIDIA GPU Operator in a Kubernetes-based AI factory?

  1. To automate deployment and lifecycle management of GPU software components on nodes Correct answer.
  2. To provide the physical InfiniBand cabling
  3. To replace the container network interface plugin
  4. To compile training frameworks from source

Explanation

GPU Operator automates installation and lifecycle of GPU-related software such as drivers, container runtime hooks, device plugins, and monitoring on Kubernetes nodes. It does not compile frameworks, replace the CNI, or provide physical cabling.

Question 48AI Factory Deployment and Management

An administrator wants to monitor GPU temperature, utilization, and ECC errors across the cluster. Which approach is most appropriate?

  1. Collect telemetry through a centralized monitoring stack using GPU metrics exporters Correct answer.
  2. Rely on the scheduler's job logs for hardware health
  3. Log into each node manually once per month
  4. Disable ECC reporting to reduce overhead

Explanation

Centralized monitoring with GPU metrics exporters provides continuous visibility into temperature, utilization, and ECC errors across all nodes. Manual logins, job logs, and disabling ECC either miss hardware health signals or remove useful error data.

Question 49AI Factory Deployment and Management

Which statement about GPU partitioning features such as MIG in an AI factory is accurate?

  1. MIG combines multiple GPUs into one logical device
  2. MIG allows a single GPU to be partitioned into isolated instances for multi-tenant workloads Correct answer.
  3. MIG disables all GPU memory isolation
  4. MIG is a network protocol for the compute fabric

Explanation

MIG partitions a supported GPU into isolated instances with dedicated memory and compute slices, which helps serve multi-tenant or smaller workloads. It does not aggregate GPUs, act as a network protocol, or remove memory isolation.

Question 50AI Factory Deployment and Management

A new GPU node joins the cluster but jobs fail to allocate GPUs. Which troubleshooting step is most directly relevant?

  1. Verify the device plugin or scheduler integration is exposing GPU resources on that node Correct answer.
  2. Replace all fabric cables in the rack
  3. Increase the front-end network MTU
  4. Reimage the storage array

Explanation

If jobs cannot allocate GPUs, the most direct check is whether the GPU resources are being advertised to the scheduler via the device plugin or equivalent integration. Storage reimaging, cable replacement, and MTU changes address unrelated subsystems.