Dual-Homed Host on a Shared Broadcast Domain

arp_ignore and arp_announce are both 0, so Linux answers ARP requests for 192.0.2.10 and 192.0.2.11 on whichever NIC receives them (ARP flux). The upstream router therefore builds an ARP cache that maps both addresses to the MAC of ens3f0, or alternates between the two MACs as entries age out. Return traffic for a session to .11 can arrive on ens3f0 even though the session was established via ens3f1.

rp_filter=1 (strict) drops any packet whose ingress interface is not the interface the kernel would use to reach the source. For a client at 203.0.113.50 the reverse route is the default via 192.0.2.1, which resolves to ens3f0. Packets that arrive on ens3f1 fail the check and are discarded silently. Early in the connection the ARP cache happens to point at the right NIC, so the TCP handshake and SSH authentication complete. When the cache entry flips, later segments are dropped, retransmissions back off, and the session appears to freeze rather than close. NFS behaves the same way, but because it opens new TCP connections and retransmits UDP traffic, the failures look intermittent rather than constant.

The second default route did not add bandwidth: both next hops are the same router on the same segment. It only made the reverse path ambiguous and increased the chance that requests and replies use different interfaces. The correct diagnosis is an L2/L3 symmetry problem, not a bandwidth or firewall problem.

Tonight, without touching the switch:

1. Remove the redundant default. It adds no path because both next hops are 192.0.2.1 on the same L2 segment, and it makes the reverse path ambiguous:

ip route del default via 192.0.2.1 dev ens3f1 metric 100
ip route flush cache

1. Stop ARP flux so each address is answered only on the interface that owns it:

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
sysctl -w net.ipv4.conf.ens3f0.arp_ignore=1
sysctl -w net.ipv4.conf.ens3f1.arp_ignore=1

arp_ignore=1 replies only when the target address is configured on the receiving interface. arp_announce=2 makes ARP announcements use the best local address for the target, so one NIC stops advertising the other NIC's address.

1. If traffic must still be accepted on both NICs before the weekend, switch reverse-path filtering to loose mode on those two interfaces:

sysctl -w net.ipv4.conf.ens3f0.rp_filter=2
sysctl -w net.ipv4.conf.ens3f1.rp_filter=2

Loose mode checks that the source is reachable via any interface, so it tolerates asymmetry. It also weakens anti-spoofing, so scope it to these interfaces and add an ingress filter that drops RFC1918 and martian sources on the WAN side.

Persist all of this in /etc/sysctl.d/90-dual-nic.conf, not just at runtime.

Permanent fix: do not dual-home two independent addresses into the same broadcast domain. Either bond the NICs (LACP, one IP, one default) or place them on separate VLANs with source-based policy routing. If policy routing is required, add ip rule add from 192.0.2.11 table 11 and a default in table 11 via ens3f1; then each source address has a deterministic path and rp_filter can stay strict.

From a client and from the host:

• ip route get 203.0.113.50 from 192.0.2.11 and the same for 192.0.2.10 should each return one deterministic path.
• ip neigh show 192.0.2.1 should show a stable MAC per address. arping -I ens3f0 192.0.2.11 and arping -I ens3f1 192.0.2.10 should now get no reply. tcpdump -eni ens3f0 arp should not show replies for .11.
• Snapshot nstat -az before and after; IPReversePathFilter should stop incrementing during a 10-minute SSH session.
• ss -tni on a long-lived session should show retransmits stop climbing.
• Keep an SSH session open with ssh -o ServerAliveInterval=5 for 15 minutes and run a 5-minute dd over NFS; check nfsstat -c for retransmissions.

For monitoring, add synthetic checks that SSH through each address and run a command, and alert on the rate of TcpExtTCPTimeouts/TcpExtTCPRetransSegs plus any increment of IPReversePathFilter. Graph ip -s neigh for MAC flapping between the two interfaces. That catches the recurrence at the ARP layer before a user sees a frozen session.