Intermittent SSH After a Second Default Route
A Ubuntu 22.04 router has two uplinks: wan0 198.51.100.10/24 via 198.51.100.1 and wan1 203.0.113.10/24 via 203.0.113.1. It NATs 10.0.0.0/24 for 40 office clients. Originally only wan0 had a default route. During ISP maintenance, an engineer added default via 203.0.113.1 dev wan1 metric 100; the existing wan0 default also has metric 100. Both routes now appear, and ip route get 8.8.8.8 alternates. Users report HTTPS stalls and VPN drops every few minutes; new connections sometimes work. conntrack -S shows insert_failed rising. The only NAT rule is -s 10.0.0.0/24 -o wan0 -j MASQUERADE. rp_filter is 2 on all interfaces. wan0 is still flapping and the maintenance window has two hours left. You need to stabilize service without causing a hard outage and leave a design that survives the next maintenance.
1. Explain the failure mechanism. Why can an equal-metric second default route break HTTPS and VPN sessions even though ping still works? Address NAT, conntrack, and return paths.
Read the worked answer
With two default routes at the same metric, the kernel can select either nexthop for a new flow; it is not a stable primary/backup pair. If a client flow leaves via wan1, the only MASQUERADE rule matches -o wan0, so that flow is not translated as intended. Return traffic then has no matching NAT or conntrack binding, or reaches the router with a different source address than the flow expects, and the stateful firewall drops it. Existing flows can also be rehashed or moved when wan0 flaps, producing asymmetric paths. Ping may still appear fine because an ICMP flow is short, may hash to wan0, and does not need the same long-lived TCP/NAT binding.
conntrack -S showing insert_failed is a warning, not proof by itself; check nf_conntrack_count against nf_conntrack_max and look for table pressure. rp_filter=2 is loose mode, so it is not the main cause here, and setting it to 0 globally would weaken spoofing protection without fixing NAT. Confirm with ip route get, NAT counters, and packet captures on both uplinks.
2. Give the first diagnostic commands and an interim policy that keeps existing flows stable while wan0 flaps. Do not disable rp_filter globally.
Read the worked answer
Start read-only: ip -br addr, ip route show table all, ip route get 8.8.8.8, ip route get 8.8.8.8 from 10.0.0.50, ip monitor link route, conntrack -S, sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max, iptables-save -t nat or nft list ruleset, and tcpdump -ni wan0 / wan1 for one client and port 443.
Interim: collapse to one active default route while wan0 is flapping, rather than letting equal-metric ECMP choose per flow. Install the wan0 default with a lower metric and wan1 with a much higher metric, and use a link or health monitor to switch only when wan0 actually fails. Before enabling the backup path, add a matching MASQUERADE rule for wan1; otherwise failover will blackhole traffic. Do not flush the entire conntrack table during business hours; that kills every flow. Accept that a real uplink change will break existing connections pinned to the old public address, and communicate that. Verify with ip route get from several client sources and a single default in ip route show default.
3. What persistent design would you choose for 40 clients, and why is simple ECMP plus one NAT rule a poor fit?
Read the worked answer
Use an active/backup design with a health-checked default route and explicit NAT for each egress. If both uplinks must be used, policy routing with connection marks can keep a flow on the same uplink: mark new connections in mangle, save the mark with CONNMARK, restore it on subsequent packets, and route marked traffic through table 100 or 200. That is more complex than ECMP and still cannot preserve a flow across a real uplink failure unless the public address is stable. For 40 office clients, the simpler and more reliable option is one primary ISP plus a backup activated only by a monitor, with a documented brief interruption for new and existing sessions.
Avoid equal-metric ECMP with a single -o wan0 MASQUERADE rule: it distributes flows onto a path that has no correct NAT, has no conntrack state synchronization, and makes failures depend on hash. Also monitor from outside, not just link state, because a carrier can be up while upstream forwarding is broken. Schedule route changes through change control and test failover during a window.