Two WANs, Conntrack Full, and One-Way VoIP Audio

Two independent faults are present.

Conntrack exhaustion. 65,000 entries against a 65,536 maximum means the table is effectively full. New flows hit nf_conntrack: table full, dropping packet and are discarded before NAT, so HTTPS feels slow because SYNs are retried. The 22,000 entries in FIN_WAIT or with 600-second UDP timeouts are the aggravating factor: each VoIP call creates many short RTP 5-tuples, and a 600-second UDP timeout keeps each entry for ten minutes after the call ends.

Return path and strict reverse-path filtering. A mark only helps if it is applied before the route lookup and if the reply follows the same WAN. Return RTP arriving on wan0 for a flow that left via wan1 is dropped by rp_filter=1: the kernel looks up the route to the provider's source address and finds wan1, so ingress on wan0 fails the check. That produces one-way audio - outbound RTP leaves, return RTP is silently discarded. If signaling and media are marked differently, or signaling is unmarked, the provider can also send media to the wan0 public address while the media stream left via wan1, which makes the mismatch systematic rather than random.

The mark rule does not fix this because the failure is on ingress filtering and flow symmetry, not on the outbound route selection alone.

Plan in order:

1. Size conntrack from RAM. An entry is roughly 300 bytes, so 262,144 entries is about 80 MB:

sysctl -w net.netfilter.nf_conntrack_max=262144
sysctl -w net.netfilter.nf_conntrack_buckets=65536
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_fin_wait=30
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
sysctl -w net.netfilter.nf_conntrack_udp_timeout=30
sysctl -w net.netfilter.nf_conntrack_udp_timeout_stream=120

Lower UDP timeouts free RTP entries quickly; RTP is continuous, so 30/120 seconds is safe. Persist in /etc/sysctl.d/.

1. Mark in prerouting, not output, so forwarded traffic is covered, and save the mark to connmark so replies keep the same WAN:

nft add rule inet fw4 prerouting udp dport 5060 meta mark set 0x1
nft add rule inet fw4 prerouting udp dport 16384-32767 meta mark set 0x1
nft add rule inet fw4 prerouting ct state established,related meta mark set ct mark

Then ip rule add fwmark 0x1 table 101 with a default via 198.51.100.1 in table 101.

1. Pin each call to one public address: SNAT marked traffic out of wan1 to 198.51.100.2, and keep signaling and media in the same mark so the provider sees one source IP. Provide the symmetric fallback via wan0, but never a pool.

2. Relax reverse-path filtering on the WAN interfaces (net.ipv4.conf.wan0.rp_filter=2, same for wan1) because multihoming is inherently asymmetric. Loose mode still requires the source to be routable; compensate with explicit ingress rules dropping RFC1918 and martian sources on both WANs.

3. Do not rely on nf_conntrack_helper for SIP; it is deprecated and breaks with unmarked media. Let the provider's SBC anchor media or run a SIP proxy that pins it.

Verify with counters and a test call at peak:

• conntrack -C and cat /proc/sys/net/netfilter/nf_conntrack_count should stay well below the new maximum, and the dmesg table-full message should stop.
• conntrack -L -p udp --dport 16384-32767 should show entries with mark=1 and a consistent reply tuple; check conntrack -L -p udp --dport 5060 as well.
• nft list ruleset with counters: confirm the prerouting marks increment for forwarded RTP, not just locally generated packets.
• tcpdump -ni wan1 udp portrange 16384-32767 and the same on wan0 during a test call; both directions must use one WAN.
• nstat -az | grep -i conntrack for NfConntrackDrop and NfConntrackEarlyDrop, and ip -s link show wan0 for drops.
• Watch dmesg -w through the peak, then place a real test call and check audio in both directions.

If return RTP still arrives on the wrong WAN but is no longer dropped, the remaining issue is signaling/media mark alignment, not filtering.