OpenStack DVR SNAT Exhausts Conntrack During Batch Window
An OpenStack Wallaby cloud has three controllers and eight compute nodes. Neutron uses ML2/OVS with VXLAN, DVR enabled, and distributed SNAT disabled. A tenant runs 200 VMs across four computes. During a nightly batch window, about 3% of new outbound TCP connections from VMs fail with SYN timeouts, while established sessions continue. The tenant has two routers, and their SNAT namespaces run on two computes. Each compute has 64 GB RAM and nf_conntrack_max=65536; nf_conntrack_count in both SNAT namespaces sits at 64k during the window. The VMs SNAT to 198.51.100.10. The operator sees nf_conntrack: table full, dropping packet in dmesg. The tenant cannot reduce VM count, and the business requires no more than 0.1% new-connection failures. The operator can change Neutron settings, add computes, or enable distributed SNAT, but must keep the same external IP.
1. Explain why established sessions survive while new connections fail, and which node's conntrack table matters.
Read the worked answer
Established sessions already have conntrack entries and are forwarded by the existing state. New connections need a new entry; when the table is full, the kernel drops the SYN. The relevant table is in the SNAT namespace on the compute that hosts that router's SNAT function, not necessarily the source VM's compute. Check conntrack -C and sysctl net.netfilter.nf_conntrack_max inside each SNAT namespace, plus dmesg timestamps. The fact that two SNAT namespaces are at 64k explains why only part of the tenant fails: each namespace has its own limit.
2. Evaluate the three options - raising conntrack limits, adding computes, or enabling distributed SNAT - against the constraints.
Read the worked answer
Raising nf_conntrack_max is the fastest mitigation, but it only buys headroom: memory use rises, and the namespace still has a hard ceiling. Adding computes does not automatically spread a given router's SNAT state; it only helps if routers are rescheduled or new routers are placed. Enabling distributed SNAT is the structural fix because each compute handles SNAT for its local VMs, splitting conntrack state and reducing the blast radius. It must keep the same external IP and preserve ARP/L2 reachability, so test upstream router behavior. A combined approach is best: enable distributed SNAT, raise the per-namespace max and bucket count, and monitor per-compute counts.
3. Give a migration plan that meets the 0.1% failure target and can be rolled back.
Read the worked answer
Pilot on one tenant with the same external IP. Enable distributed SNAT in a canary compute group, verify the upstream router sees the same IP with correct ARP, and confirm each compute's SNAT namespace stays below 50% of nf_conntrack_max. Tune max to at least 262144 with proportional buckets, and review TCP timeouts so idle sessions do not pin entries. Run the nightly batch at 1.5x expected load, measure new-connection failure rate against 0.1%, then roll out rack by rack. Keep a rollback path to centralized SNAT and a dashboard for nf_conntrack_count, nf_conntrack_max, and drops.