Consolidating a Hospital AI Platform Before Pilot Expansion
Northvale Health System runs a private AI platform for radiology and clinical documentation. The environment has two Nutanix clusters: a 3-node management cluster running Prism Central and NKP management components, and a 6-node workload cluster with 24 NVIDIA L40S GPUs. Data services include Nutanix Files for imaging exports and Nutanix Objects for de-identified training corpora. Six months ago, a team deployed a GPT-in-a-Box-based inference endpoint on the workload cluster and began sharing the same NKP namespace with data-science notebooks and nightly fine-tuning jobs. Since then, the clinical documentation endpoint has missed its 99.5% monthly availability target twice, GPU memory errors appear during training, and the storage team reports hot spots on the NFS exports during checkpointing. The security office requires PHI to remain on-premises and wants evidence that research workloads cannot affect clinical inference. The platform team has budget for four more GPU nodes but must justify the purchase with a design, not just capacity.
1. The CIO wants to add four GPU nodes to the existing workload cluster. As the architect, what would you change in the platform design before approving the purchase? Address cluster and namespace placement, GPU scheduling, and data paths.
Read the worked answer
Adding nodes to the same shared namespace would preserve the blast radius, so I would fix placement first. Put clinical inference endpoints in a dedicated production namespace on the workload cluster and move notebooks and fine-tuning into separate research namespaces with their own service accounts and quotas. Create at least two GPU node pools: one for latency-sensitive inference with N+1 headroom, and one for training with taints that only training pods tolerate. Use priority classes so inference pods preempt batch work, and anti-affinity so replicas do not land on the same host. Keep Prism Central and NKP management components on the management cluster, and do not run production inference there.
For data, stop checkpointing to the same NFS exports that serve imaging. Use Nutanix Objects as the durable checkpoint and model-artifact store, with local NVMe or a dedicated Nutanix Volumes-backed scratch path for active training writes, then copy completed checkpoints to Objects. Put imaging exports on a separate Files share or project with Flow policies limiting research access. Only after these changes would I add GPU nodes, because the purchase should increase isolated inference capacity rather than deepen contention. The key judgement is that average GPU utilization is not the problem statement; the problem is shared failure domains and uncontrolled scheduling.
2. What evidence would you present to show whether the problem is capacity, contention, or storage I/O? Name the specific signals you would collect and how you would interpret them.
Read the worked answer
I would build an evidence pack around SLO burn and queueing, not a single utilization number. Collect per-GPU utilization and memory used versus reserved from DCGM, XID and ECC errors, and per-pod GPU requests versus actual use. From NKP, collect pending GPU pods, scheduling latency, preemption events, namespace quotas, and pod restarts. For inference, collect p50/p95/p99 latency, queue depth, tokens per second, model load time, and error rate. For storage, collect NFS latency and IOPS during checkpoint windows, Objects request latency, and checkpoint duration.
Interpretation matters: if GPUs show 38% average utilization but p95 latency and queue depth rise during training, the constraint is contention or memory headroom, not raw GPU count. If pending pods exist while GPUs report free memory, the scheduler is matching full-GPU requests, not actual use. If XID errors cluster on hosts running both training and inference, co-location is the risk. If NFS latency spikes only during checkpointing and inference latency follows, storage I/O is the trigger. I would show a timeline correlating training start, checkpoint start, GPU memory errors, and missed SLOs, plus a capacity model that includes peak concurrency and N+1 headroom.
3. If the purchase is deferred, what alternatives would you propose, and what risks would you attach to each?
Read the worked answer
I would offer a ranked set of non-hardware options with explicit risks. First, enforce scheduling separation and quotas so training cannot consume inference GPUs; this is low risk and should be done regardless. Second, introduce quantization or a smaller distilled model for the clinical documentation endpoint, validated against a clinical evaluation set; the risk is quality regression and the need for clinician sign-off. Third, add continuous batching and response caching at the inference endpoint to raise throughput per GPU; the risk is increased tail latency if batching is too aggressive. Fourth, move fine-tuning to a scheduled overnight window with checkpointing to Objects; the risk is longer experiment turnaround and missed deadlines. Fifth, use vGPU or time-sliced sharing only for research notebooks, never for production inference; the risk is noisy-neighbor variability and weaker fault isolation. I would not present these as equivalent to capacity. They buy time, but if the SLO model shows peak inference demand plus N+1 exceeds current GPU memory, deferral accepts a known availability risk and should be recorded as such.