GPU Sharing for a Regulated Quant Research Group
Meridian Capital's quant research group has 22 data scientists and a production risk engine. The AI platform runs on AHV across two Nutanix nodes, each with four NVIDIA A100 80GB GPUs, for eight GPUs total. Compliance requires that each research VM's model code and market-data cache remain isolated at the hypervisor boundary; container time-slicing was rejected in the last audit because GPU memory and fault domains were shared. The team wants interactive notebooks during the day and multi-GPU fine-tuning at night. Production inference needs deterministic latency and cannot tolerate another tenant resetting a GPU. Current practice is full GPU passthrough to four large VMs, which leaves four GPUs idle most of the day while users queue. The infrastructure team proposes NVIDIA vGPU across all eight GPUs to raise utilization. The security team asks whether vGPU satisfies the isolation requirement and how to prevent a research workload from affecting production.
1. Recommend a GPU allocation model for research and production. Compare passthrough, vGPU, and time-slicing against the compliance and latency requirements.
Read the worked answer
I would not use a single sharing model across all eight GPUs. vGPU is the right fit for interactive research because it preserves a hypervisor-level VM boundary and partitions GPU memory, which addresses the audit finding that container time-slicing shared memory and fault domains. Time-slicing inside containers is unsuitable here because it does not provide the required VM isolation and can let one process affect another's latency. Passthrough remains the better choice for production inference and multi-GPU fine-tuning because it gives the VM exclusive use of a GPU, avoids vGPU scheduling overhead, and provides the most deterministic performance.
A practical split for eight A100 80GB GPUs is two GPUs passthrough to production inference, two passthrough to a dedicated training VM or node pool, and four GPUs configured for vGPU profiles matched to research workload memory. Research VMs get one or more vGPUs based on model size, but I would not overcommit GPU memory. If multi-GPU fine-tuning needs NVLink-scale performance, it belongs on passthrough GPUs, not vGPU. The judgement is that higher average utilization is not worth weakening production isolation. I would also separate production and research onto different AHV hosts or clusters where possible, because a host-level GPU fault can still affect all workloads on that host.
2. How would you configure AHV and NKP so production inference cannot be displaced or slowed by research workloads?
Read the worked answer
At the AHV layer, I would dedicate hosts or GPUs to production and avoid mixing passthrough and vGPU profiles on the same physical GPU. Install the NVIDIA vGPU manager and driver versions validated for the AHV and AOS release, and document the support matrix. For production VMs, use GPU passthrough with anti-affinity so the two inference VMs do not share a host failure domain. Reserve CPU and memory for production VMs and monitor steal time and GPU XID errors.
On NKP, put production inference in a dedicated node pool with taints and labels, and give only production pods the matching tolerations and node affinity. Apply ResourceQuotas and LimitRanges per namespace, and use PriorityClasses so production pods cannot be preempted by research jobs. Use PodDisruptionBudgets for inference replicas and admission policies that reject GPU requests without the production toleration. Research notebooks should run in a separate namespace and node pool, with quotas that prevent a single user from claiming all vGPUs. If production inference runs in VMs rather than pods, keep it out of the shared NKP scheduling path entirely. The design principle is physical or hypervisor-level separation for production, and policy-enforced separation for research.
3. What licensing and lifecycle considerations would you raise before implementing vGPU?
Read the worked answer
I would raise licensing, compatibility, and operational ownership before any build. AI workloads using vGPU generally require NVIDIA AI Enterprise or Virtual Compute Server entitlements, licensed per GPU or per vGPU depending on the NVIDIA program; the team must confirm entitlement before enabling profiles. Passthrough AI workloads may also require NVIDIA AI Enterprise support depending on the support model, so I would not assume passthrough is unlicensed. I would validate that the A100, the chosen vGPU profiles, the AHV release, and the guest drivers are all in the supported matrix.
Lifecycle is the bigger risk. vGPU introduces a hypervisor-level NVIDIA manager that must be patched with AHV, and guest drivers must stay aligned with NKP GPU Operator components. I would define a maintenance window, a rollback path, and a test group before touching production hosts. I would also track GPU EOL and driver support dates, because a working pilot can become unsupported after an AOS upgrade. Finally, I would document the allocation map: which physical GPU serves which profile, which VM owns it, and which team pays for the entitlement. Without that map, utilization will drift back to large passthrough VMs and the original queueing problem will return.