SR-IOV vSRX Service Chain Starves One DPDK Queue
An operator runs a three-site NFV service chain on KVM hosts. Each host has two 25 GbE Intel E810 NICs, 512 GB RAM, and sixteen 1 GB hugepages. The chain is tenant VM to vSRX VNF to tenant VM. vSRX 22.4 uses DPDK 21.11 with four lcores pinned to physical cores 4 through 7, and each SR-IOV VF has four RX/TX queues. The underlay MTU is 9000; tenant MTU is 1500. At 8 Gbps bidirectional, vSRX reports 2% packet loss and rising rx_missed on VF queue 0, while the other three queues show near-zero drops. The VF is on NUMA node 1, but the hugepages are allocated on node 0. Flow steering uses RSS only, and the return path is handled by a single lcore on core 4. The service chain is symmetric on paper, but the return path uses a different lcore and queue. The operator wants 10 Gbps with less than 0.01% loss and asks whether to add queues, add lcores, or redesign the chain.
1. Which two observations most strongly point to host and queue placement rather than raw link capacity, and what would you inspect first?
Read the worked answer
The two strongest clues are: (1) drops concentrate on VF queue 0 while queues 1-3 are nearly clean. That is a per-queue servicing or steering problem, not a 25 GbE link limit; 8 Gbps is far below line rate. (2) The VF is on NUMA node 1 while the hugepages are on node 0. DPDK packet buffers and descriptor rings then cross NUMA, raising memory latency and reducing the lcore's ability to refill RX descriptors. The single return-path lcore on core 4 also means one core can become the bottleneck even when other lcores are idle.
First inspect ethtool -S on the VF for per-queue rx_missed and rx_no_buffer, DPDK rte_eth_stats and rte_flow rules, lcore-to-queue mapping, and hugepage allocation per NUMA node. Confirm whether RSS spreads flows or whether one flow is pinned to queue 0. Do not add queues until queue-to-lcore and NUMA placement are understood.
2. The operator wants to double the VF queues from four to eight. Evaluate that proposal and give a safer change sequence.
Read the worked answer
Doubling queues from four to eight is not a safe first move. More queues add more RX rings and more polling or interrupt work; if queue 0 is already starved and the return path is on one lcore, eight queues can increase contention and cause out-of-order delivery. A safer sequence is: move the VF and hugepages to the same NUMA node; isolate cores 4-7 from the host scheduler; assign one RX/TX queue pair per lcore and keep both directions of a flow on the same lcore with symmetric RSS or flow director; increase RX/TX descriptors; verify VF trust, promiscuous mode, and VLAN settings; then load-test at 8 Gbps and 10 Gbps. Only after per-lcore utilization and rx_missed are balanced should you raise queue count. Validate with 1500-byte tenant traffic and 9000-byte underlay.
3. What design changes would you make to reach 10 Gbps bidirectional at under 0.01% loss while keeping the tenant MTU at 1500?
Read the worked answer
I would redesign placement before adding capacity. Put the VF, hugepages, and vSRX lcores on the same NUMA node; dedicate lcores to RX, TX, and return-path processing; use flow director or symmetric RSS so a flow is not split across lcores; size descriptors for 10 Gbps bursts; and keep tenant MTU 1500 with underlay 9000 so no fragmentation occurs. The service chain must be symmetric: ingress and egress of the same flow should traverse the same VF and queue pair. Add monitoring for per-queue drops, lcore cycles, and hugepage locality. If 10 Gbps still cannot be met with one vSRX instance, scale horizontally with multiple VFs and ECMP service-chain nodes, not by adding queues to a single starved lcore.