Tail-Latency Regression After a NUMA and Power-Management Change

The evidence constrains the hypothesis space. Throughput is unchanged and p50 actually improved slightly (1.2 → 1.1 ms). A CPU that is simply too slow inflates the median as well as the tail, so the 'buy a faster SKU' proposal is not supported by the data. The regression is tail-selective, which points to something that only happens when a core is idle, when memory is remote, or when work migrates.

Ranked candidates:

1. Deep C-state exit latency (BIOS). Exit latencies of 100–500 µs per wakeup are typical for C6/package C-states. That alone is far short of 40 ms, but it is paid on exactly the requests that arrive at a momentarily idle core, so it is concentrated in the tail.
2. Cross-node memory. All NIC interrupts land on node 0 while the heap spans four nodes. Descriptors, buffers and the application's data structures are therefore frequently remote, costing roughly 1.5–2× local memory latency plus cross-node cache-line ping-pong.
3. Sub-NUMA clustering split co-location. What used to be one node is now two, so thread/IRQ pairs that were previously co-located may now be separated - a silent topology change, not a clock change.

Why small per-message penalties produce a 40 ms tail: at high utilization, queueing delay grows approximately as 1/(1−ρ). A 10% service-time penalty on a system running at ρ ≈ 0.9 can roughly double mean wait, and the same amplification applies to every rare slow path. The combination of (1)–(3) compounds inside the queue.

Revert first: the C-state policy, because it is the cheapest, most reversible and most tail-selective of the three. Do it on a canary host, not fleet-wide, and hold the kernel and NUMA settings constant so the effect is attributable.

Split the plan into 'is it the idle state', 'is it memory placement', and 'is it the application', and make each test isolating.

Idle-state residency. Run turbostat and read /sys/devices/system/cpu/cpu*/cpuidle/state*/time on the canary. If the p99 spike correlates with cores that spent time in C6 and p99 drops when you boot with processor.max_cstate=1, hypothesis 1 is confirmed. Keep an eye on power and temperature: capping C-states raises package power noticeably at this socket count.

Memory placement. Use numastat -m, /proc/<pid>/numa_maps and uncore counters (perf stat -e node-loads,node-load-misses,node-stores) to quantify the remote fraction. Then A/B against numactl --cpunodebind=0 --membind=0; if local-only placement recovers most of the tail, hypothesis 2 is confirmed. Verify that irqbalance is not moving queues, and read /proc/interrupts per queue to see the current affinity.

Where the time is spent. Instrument the service to separate queue wait (time between the socket becoming readable and the handler starting) from service time. If queue wait carries the 40 ms, the problem is scheduling, wakeup and memory; if service time carries it, look for allocation or GC events.

Rule out confounders. Check whether the kernel upgrade changed the default idle governor, THP mode or NIC interrupt moderation, because any of those would produce the same signature. Record all of these as part of the canary baseline so the post-change comparison is defensible.

Order matters: change one variable at a time and keep a canary at each step.

1. Cap C-states to C1 for the latency-critical cores (BIOS setting plus intel_idle.max_cstate=1 as a belt-and-braces kernel parameter). Re-measure p99 and p99.9. Expect the largest single improvement here.
2. Co-locate interrupts and workers. Pin each NIC queue to a CPU on the same node as the threads that read it, either with explicit SMP affinity or by running one service instance per node under numactl --cpunodebind=N --membind=N with its own queue set. This removes the remote-memory penalty without touching the code.
3. Bound the heap per node. Replace the default cross-node reservation with per-instance local allocation so that no instance straddles a node boundary.
4. Only if compaction stalls appear, set THP to madvise; do not change it speculatively, because it is easy to blame and hard to attribute.

On the CPU purchase: reject it for now, and say why explicitly - throughput is identical and the median latency did not regress, so the CPU is not the constraint; a faster core would not fix a wakeup-latency and memory-locality problem and would cost a maintenance window plus capital. Revisit only if, after steps 1–4, single-core utilization on the busiest cores exceeds roughly 70% at peak and perf shows the cores themselves as the bottleneck.

Close the loop: add a synthetic canary workload that reproduces the 120 k msg/s rate, and gate future BIOS or kernel upgrades on it, because this regression was invisible to functional tests.