The tenant whose guest says 35% idle but whose queries take 3.5 seconds
An internal platform team runs a three-host private cloud and sells a standard 16-vCPU / 64 GB VM. Each host has two 32-core CPUs with hyperthreading enabled and 512 GB of RAM. Host A currently carries 9 of the standard VMs. The cluster's CPU allocation ratio is configured at 1:8. Memory on Host A is 96% allocated, and the hypervisor is reporting ballooning activity on three of the nine VMs. A tenant running a database on Host A reports that queries which normally complete in 400 ms now take about 3.5 s starting around 02:00 each night. Inside the guest, top shows roughly 35% idle and the application is not reporting lock waits. Running vmstat in the guest shows the steal-time column fluctuating between 12 and 40. A new tenant was onboarded last week; it runs a nightly single-threaded Java batch job pinned at 100% on one core, plus a second job that scans a large in-memory index.
1. Explain what the steal-time values mean and why the guest can report 35% idle while a query slows from 400 ms to 3.5 s.
Read the worked answer
Steal time is the fraction of time a virtual CPU was runnable - it had work to do - but the hypervisor scheduled a different vCPU on the physical core instead. From inside the guest it does not appear as load: the guest cannot see the work it never got to run. That is precisely why top can show 35% idle and the application can still be slow. The guest's view is a vCPU's view, not the physical host's.
The number matters in magnitude, not just existence. Small single-digit steal is normal noise on a shared host. Between 12 and 40, the database VM is losing up to 40% of its scheduled CPU time at unpredictable moments. A 400 ms query becomes 3.5 s not because it needs 8.75x more CPU, but because every time the vCPU is descheduled, latency-sensitive work stretches - a lock held slightly longer, a buffer pool miss that suddenly costs a full scheduler wait, a network round-trip queued behind a descheduled thread. Latency compounds; it does not scale linearly.
The 35% idle figure is also consistent with a database that is not CPU-saturated in aggregate but is bursty. A 16-vCPU VM handling a query workload typically has a few vCPUs busy and the rest idle. The busy ones are exactly the ones being stolen from, and the idle ones drag the average up.
The 02:00 onset is the strongest clue: it aligns with the new tenant's nightly batch. This is not a capacity problem in aggregate - it is a co-scheduling problem on one host.
2. Evaluate whether the 1:8 CPU allocation ratio is the cause. Support your answer with the actual numbers on Host A.
Read the worked answer
The 1:8 ratio is the configured ceiling, not what Host A is actually doing, and conflating the two is the classic mistake here. The real oversubscription is computed from what is allocated, not from what the policy permits.
Host capacity. Two 32-core CPUs = 64 physical cores. With hyperthreading, 128 logical CPUs. Two caveats on the 128: hyperthread siblings share execution units, so a vCPU on a sibling pair does not deliver a full core's throughput, and the security guidance for multi-tenant hosts is generally to avoid scheduling untrusted tenants across siblings at all. Treat 64 as the honest number for capacity planning and 128 as an optimistic ceiling.
Host A allocation. 9 VMs × 16 vCPU = 144 vCPU. Against 64 physical cores that is 2.25:1. Against 128 logical CPUs it is 1.125:1 - technically under 1:1 on logical CPUs.
So the 1:8 policy is not being exercised anywhere near its limit on this host; the host is barely oversubscribed on paper. Yet steal time is 12-40 and memory is at 96%. That tells you the problem is not the ratio setting but the shape of the allocation: the VMs are provisioned at 16 vCPU each because that is the product SKU, not because any of them needs 16. The database VM probably needs 4-6 busy vCPUs. Nine 16-vCPU VMs on 64 real cores has almost no headroom for the moment when several tenants' busy vCPUs land on the same core.
Conclusion. Blaming the 1:8 ratio would lead to lowering it, which changes nothing on this host and only limits future growth. The correct levers are vCPU right-sizing, CPU resource reservation for the latency-sensitive VM, and anti-affinity so the batch tenant and the database tenant stop sharing cores at 02:00.
3. The ballooning report and the 96% memory allocation are on the same host. Explain the mechanism and what you would change, and why memory pressure can make a CPU diagnosis misleading.
Read the worked answer
Mechanism. Ballooning is the hypervisor reclaiming guest memory under host pressure. A balloon driver inside the guest is told to inflate, which pins guest-physical pages; the hypervisor then swaps or discards those pages and the guest's free memory shrinks. On Host A, 9 × 64 GB = 576 GB is allocated against 512 GB physical, so the host must reclaim something - ballooning on three VMs is the expected consequence, not an anomaly.
Why it corrupts the CPU diagnosis. The database VM's buffer pool is now partially paged out. A query that used to hit memory now takes a major page fault, which is an IO operation with the vCPU blocked. Meanwhile that vCPU may also be stolen from. The two effects are indistinguishable from inside the guest using top alone: both show up as slow queries with low apparent CPU utilisation, and the steal figure may understate the real damage because a vCPU waiting on a page fault is not runnable and therefore is not counted as stolen. If you only chase the steal number, you will fix half the problem and the 02:00 slowdown will persist.
What to change, in order:
- Give the database VM a memory reservation so its working set cannot be ballooned. This is the single highest-value change and it is immediate.
- Reduce the SKU or the count on Host A so allocation fits within 512 GB with headroom for the hypervisor itself - 96% allocated is past the point where reclaim is safe.
- Right-size the batch tenant's memory footprint; a full index scan over an in-memory structure may be bloating its working set and triggering the reclaim in the first place.
- Apply anti-affinity between the database VM and the batch tenant, so the nightly job and the latency-sensitive workload stop competing for the same core and the same memory.
Check guest-level swap-in/swap-out and the hypervisor's balloon and swap counters alongside the steal figure, because only those distinguish "starved of CPU" from "starved of memory and then starved of CPU".