Fixing GPU Scheduling and Autoscaling on NKP

The apparent contradiction is explained by request-based scheduling. Kubernetes and the NVIDIA device plugin allocate whole GPUs based on pod requests, not live utilization. If each model pod requests one full GPU, a node with four GPUs can host only four such pods even when each uses 8GB. The 12 replicas from the unquoted namespace can consume all advertised GPUs and leave other teams pending. Meanwhile, GPU utilization stays low because those pods are memory-light, and CPU-only moderation pods land on GPU workers because there is no taint, consuming memory and possibly causing eviction or scheduling pressure.

I would collect node allocatable GPU counts, pod GPU requests, pending pod events, namespace quotas, and scheduler messages. I would also check whether the HPA is scaling on CPU utilization rather than GPU or request-queue metrics, because CPU-based scaling can over-provision GPU replicas. The evidence should show a mismatch between requested and actual GPU memory, plus a lack of tenant quotas. The judgement is that adding a third GPU node would temporarily relieve the symptom while preserving the same wasteful request pattern, so it should not be approved alone.

I would create dedicated GPU node pools with taints and labels, and allow only GPU workloads with matching tolerations to run there. CPU-only services stay on CPU pools. Each tenant namespace gets a ResourceQuota and LimitRange for GPU requests, plus PriorityClasses so latency-sensitive inference can preempt batch or low-priority work. For small models that use 8GB, I would evaluate MIG or GPU time-slicing, but only with clear isolation and performance expectations; MIG gives hardware partitioning on supported GPUs, while time-slicing improves density but weakens fault and latency isolation.

Autoscaling should be driven by service demand, not CPU. Use KServe or HPA with custom metrics such as request queue depth, concurrency, or p95 latency through a Prometheus adapter. Set min and max replicas per service, and use scale-to-zero only for tolerant workloads because cold starts hurt latency. A cluster autoscaler can add GPU nodes when pods are unschedulable, but it must understand the GPU node pool and taints. I would also define PodDisruptionBudgets and topology spread constraints for critical inference. The goal is to right-size requests first, then add density features, then add nodes only for justified headroom.

I would upgrade in a staging cluster first, with the same GPU Operator, device plugin, driver, and KServe versions. Before production, verify the NKP release is compatible with the AHV and AOS versions and with the NVIDIA driver and GPU model. The upgrade plan should cordon and drain workers one at a time, respecting PodDisruptionBudgets and ensuring enough GPU capacity remains for replicas during each drain. Control-plane components upgrade first, then worker node pools, with GPU workers last or in a separate wave.

I would confirm that inference services have at least two replicas and anti-affinity where possible, and that model artifacts are in Objects so a rebuilt node can pull them without internet. Rollback must be tested: keep the previous NKP version artifacts, node images, and GPU Operator configuration, and know whether rollback requires reimaging workers. During the window, monitor pending GPU pods, endpoint error rate, p95 latency, and XID errors. If a canary service degrades after the first GPU worker returns, stop the wave and roll back that pool rather than continuing. The upgrade is successful only when scheduling, GPU discovery, and model serving all pass a post-upgrade smoke test.