Throughput Ceiling on a Dual-Socket Flow Analyser
A flow-analytics appliance runs on a dual-socket server: 2 × 16 cores, four NUMA nodes (two per socket, 8 cores each), 256 GB DDR4-3200. A 40 Gb/s NIC in a socket-0 PCIe slot feeds eight RX queues via RSS. Each of N worker threads processes flows and increments a counter held in a shared array declared as uint64_t hits[MAX_FLOWS], where consecutive flow slots are 8 bytes apart. Throughput scales linearly to 10 threads (24 Gb/s), plateaus at 22 Gb/s, and falls to 19 Gb/s at 32 threads. CPU utilisation never exceeds 62%, and the NIC reports 4% RX drops. perf shows elevated LLC misses with offcore-response stalls; /proc/interrupts shows every RX queue's interrupts landing on cores 0–7; numastat shows the 4 GB flow table almost entirely on node 0. Workers are unpinned. The team's first proposal is to purchase a faster NIC.
1. Two mechanisms are consistent with the plateau: false sharing on the counter array, and cross-NUMA access to the flow table and packet buffers. Which is dominant, and what experiment separates them?
Read the worked answer
Both are real, but they leave different signatures, so the question is which one sets the 22 Gb/s ceiling.
False sharing fits the counter array because eight 8-byte slots share one 64-byte line. Eight threads incrementing neighbouring slots invalidate that single line continuously; the line migrates between sockets and the traffic is bounded by the coherence interconnect, not by core count. That explains the plateau at 62% CPU and the fall at 32 threads, when more threads contend for the same few lines.
Cross-NUMA fits the topology: the NIC sits on socket 0, IRQs land on cores 0–7, so packet buffers and the first-touch flow table are allocated on node 0. Node-1 workers pay remote-latency reads for every lookup, and remote traffic crosses UPI, which is also a shared, finite resource.
Separate them with two controlled experiments. First, pad each counter to a 64-byte boundary (or use per-thread counters summed at read time) and re-measure: a large recovery implicates false sharing. Second, separately pin all workers and IRQs to node 0 and re-measure: a large recovery implicates NUMA placement. Running them one at a time keeps attribution clean; running both at once gives a number but no diagnosis.
2. Write the measurement plan you would run before changing code, naming the specific evidence each step produces.
Read the worked answer
The plan must produce mechanism-specific evidence, not just a bigger throughput number.
- Baseline with
perf stat -e LLC-loads,LLC-load-misses,offcore_response.*at 10, 16 and 32 threads to show whether the miss rate scales with thread count while IPC falls. perf c2c record/reportacross workers. HITM (modified-line) records pinpoint the exact cache lines being ping-ponged; if the counter array appears, false sharing is confirmed by address, not by inference.numastat -p <pid>and/proc/<pid>/numa_mapsto quantify what fraction of the flow table and packet buffers is remote, plusmpstat -P ALLper-node CPU and/proc/interruptsdeltas to confirm IRQ concentration.- Read the NIC drop counters alongside software counters to confirm drops are a consequence of the software ceiling, not the cause.
- Only then run the A/B interventions: padded counters, and node-0 pinning, each in isolation, restored between runs.
Record variance across at least three runs; a 2 Gb/s difference is inside run-to-run noise on this class of machine, and the plateau question is about a 6 Gb/s gap, so noise must be bounded before conclusions are drawn.
3. Recommend a remediation sequence, with the expected effect and the risk of each step.
Read the worked answer
Order the fixes by cost, reversibility and diagnostic value.
Step 1: pad counters to one cache line each, or better, keep per-thread counters on the thread's own stack and aggregate on read. Cost is a small code change; risk is low, and the read path becomes slightly more complex. Expect the largest single gain if perf c2c implicated those lines.
Step 2: align IRQ affinity and worker placement to the NIC's node, or use two NICs, one per socket. At minimum, allocate the flow table with first touch from the thread that will own each shard, so the table is partitioned per node rather than concentrated on node 0. Risk: pinning reduces scheduler freedom and can hurt tail latency during bursts; verify with load tests, not averages.
Step 3: shard the flow table by hash so each worker owns a private, node-local partition, and batch counter updates to amortise coherence traffic.
Do not buy a faster NIC first. The evidence says the CPUs are 38% idle and the bottleneck is memory-system behaviour; a 100 Gb/s NIC would deliver drops, not throughput, and would consume the budget that the real fix needs. Re-run the baseline after each step so the remaining ceiling is re-attributed.