LPIC-3 Virtualization and Containerization: NUMA Locality Regression on a 24-vCPU KVM Database Guest

NUMA Locality Regression on a 24-vCPU KVM Database Guest

A latency-sensitive PostgreSQL VM named db-prod runs on a KVM host with two NUMA nodes. Node0 owns CPUs 0-15 and 128 GiB; node1 owns CPUs 16-31 and 128 GiB. The VM has 24 vCPUs, 96 GiB RAM, virtio-blk disks, and host-passthrough CPU. No vCPU pinning or NUMA tuning is configured. After a host kernel upgrade, pgbench throughput on db-prod is 28% below bare metal and p99 commit latency has doubled. numastat -c qemu-kvm reports 71 GiB allocated on node0 and 25 GiB on node1, while numastat -m shows node1 has 40 GiB free. virsh vcpuinfo db-prod shows vCPUs scheduled on both nodes with frequent migrations. The host also runs six small 2-vCPU VMs. The business owner requires the database to stay on this host and allows a maintenance window of no more than five minutes.

1. Which host and guest measurements would you take before editing the domain XML, and what would each tell you?

Read the worked answer

Collect placement and pressure data first; the numbers in the ticket already suggest a locality problem, but you need to prove it.

virsh vcpuinfo db-prod gives the host CPU for each vCPU and the migration count. Frequent migrations plus vCPUs spread over both nodes explain run-queue latency and cache thrash.
virsh numatune db-prod shows whether a memory policy exists. With no <numatune>, QEMU follows the host default policy, so pages land wherever they were first touched.
numastat -p $(pgrep -f 'qemu.*db-prod') breaks resident memory down by node. 71 GiB/25 GiB on a guest whose vCPUs run on both nodes means roughly a quarter of memory accesses are remote for half the vCPUs.
numastat -m and /sys/devices/system/node/node*/meminfo confirm whether node0 can actually hold 96 GiB of hugepages after the other six VMs are accounted for.
virsh capabilities confirms the host topology and distances that libvirt will use.
cat /sys/kernel/mm/transparent_hugepage/enabled and grep Huge /proc/meminfo tell you whether the guest is relying on THP or explicit hugepages. This matters because the fix will reserve hugepages per node.
virsh list --all plus virsh vcpuinfo for the small VMs shows how much of node0 is already committed. Pinning is only safe if you can reserve 16 pCPUs and 96 GiB on node0.

Do not change anything until you can state the target: all guest memory and all vCPUs on one node, or two explicitly sized vNUMA cells.

2. Compare two remediation designs: shrinking the guest to 16 vCPUs pinned to node0, versus keeping 24 vCPUs with two vNUMA cells. Which do you choose and why? Include the key XML elements.

Read the worked answer

Design A shrinks the guest to 16 vCPUs on node0:

<vcpu placement='static'>16</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='0'/>
  <!-- ... vcpu 15 -> cpuset 15 ... -->
  <emulatorpin cpuset='0-1'/>
</cputune>
<numatune><memory mode='strict' nodeset='0'/></numatune>
<memoryBacking><hugepages><page size='2048' unit='KiB'/></hugepages></memoryBacking>

This requires 49,152 free 2 MiB hugepages on node0. Every vCPU and every page is local; VM exits and memory scans stay on one node. The cost is losing 8 vCPUs, so gate it on a pgbench run that proves 16 vCPUs meet the SLA.

Design B keeps 24 vCPUs but exposes host topology to the guest:

<cpu><numa>
  <cell id='0' cpus='0-15' memory='67108864' unit='KiB'/>
  <cell id='1' cpus='16-23' memory='33554432' unit='KiB'/>
</numa></cpu>
<cputune>
  <vcpupin vcpu='0' cpuset='0'/>
  <!-- vcpu 15 -> 15; vcpu 16 -> 16 ... vcpu 23 -> 23 -->
</cputune>
<numatune>
  <memory mode='strict' nodeset='0-1'/>
  <memnode cellid='0' mode='strict' nodeset='0'/>
  <memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>

I would choose Design A for this workload. PostgreSQL keeps one shared buffer pool; it does not partition that pool by vNUMA cell, so Design B still leaves a large shared region with remote accesses from whichever node does not own those pages. Design B is a fallback only if benchmarking shows 16 vCPUs cannot meet throughput.

3. What operational risks does your design create for live migration and for the five-minute maintenance window, and how do you contain them?

Read the worked answer

Static pinning and strict memory binding make the guest non-portable. virsh migrate --live will fail on a destination that cannot satisfy the exact pCPU set or does not have 96 GiB of free hugepages on the target node. Contain this by treating the cluster as a homogeneous pool and by preparing a destination-specific XML that remaps vCPUs to the target node, then using virsh migrate --live --xml dest.xml db-prod qemu+ssh://host2/system. Before any migration, check virsh freepages --cellno 0 on the target.

For the five-minute window, note that <vcpupin> can be changed on a running domain with virsh vcpupin, but changing <memoryBacking> to hugepages and <numatune> to strict mode normally needs a shutdown and start. Do not promise a live fix. The runbook should be: pre-define the tuned XML, pre-allocate hugepages, shut down the guest, start it from the tuned definition, run a short smoke test, and keep an untuned fallback XML ready. Reserve the pCPUs for db-prod by pinning the six small VMs away from node0, or the host scheduler will still place work there. Monitor virsh numatune, numastat -p, and p99 latency after the change; if remote memory persists, the pinning did not take effect.