LFCS study materials
LFCS study resources

SSH sessions hang on a dual-homed server: wrong prefix length, not a firewall

LFCS · Put your knowledge to work in context.

All case studies

SSH sessions hang on a dual-homed server: wrong prefix length, not a firewall

A Debian 12 server has two NICs: eth0 10.20.0.10/24 with default route via 10.20.0.1, and eth1 192.168.50.10 configured with a 32-bit mask (a typo made during a late-night change). A static route 192.168.0.0/16 via 10.20.0.1 exists for a legacy subnet. Clients on 192.168.50.0/24 report that SSH connections to 192.168.50.10 hang after the password prompt and eventually time out, and that ping is unreliable. tcpdump -ni any port 22 on the server shows replies to 192.168.50.25 leaving via eth0 with source address 10.20.0.10. The upstream router performs unicast reverse-path filtering. The firewall on the server is default-accept. The 10.20.0.0/24 side must keep working; you cannot change client configuration.

1. Explain why `ip route get 192.168.50.25` returns `via 10.20.0.1 dev eth0 src 10.20.0.10`, and give the commands that confirm the root cause rather than a symptom.

Read the worked answer

Because eth1 was given a /32, the kernel never installed a connected route for 192.168.50.0/24. The only route covering the client address is the static 192.168.0.0/16 via 10.20.0.1, so the lookup picks it, and the source address is chosen from the outgoing interface (eth0).

That is textbook asymmetric routing: the request arrives on eth1, the reply leaves on eth0 with a source the client's path does not expect. The upstream router's uRPF check drops those replies, so TCP connects but never progresses - which matches "hangs after the password prompt" and unreliable ICMP.

Confirm the cause, not the symptom:

ip -4 -br addr show            # look at the prefix on eth1: /32 instead of /24
ip route show                  # is there a 192.168.50.0/24 proto kernel route? there should not be
ip route get 192.168.50.25
ip route get 192.168.50.10

If ip route show has no 192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.10, the prefix is wrong. tcpdump alone only shows the symptom; the missing connected route is the cause.

2. Apply the fix live, make it survive reboot on a Debian 12 host using /etc/network/interfaces, and state what you would check on the 10.20.0.0/24 side afterwards.

Read the worked answer

Live fix (disruptive only to the affected path):

ip addr replace 192.168.50.10/24 dev eth1
ip route get 192.168.50.25      # must now be: 192.168.50.25 dev eth1 src 192.168.50.10
ip -4 -br addr show eth1

Do not delete the 192.168.0.0/16 route - once the /24 connected route exists it is more specific and wins for this subnet, and the legacy route is still needed elsewhere. Deleting it may break the other host group, which is outside your change authority.

Persist it:

# /etc/network/interfaces
auto eth1
iface eth1 inet static
    address 192.168.50.10/24

ifreload -a    # or: systemctl restart networking

Afterwards check:

ip route get 192.168.50.25
ss -tn state established '( sport = :22 )'
nstat -az | grep -i ReversePath   # should stop incrementing

Established sessions will recover on their own once replies take the correct path; if any remain stuck, they were mid-handshake during the change and can be reconnected. On the 10.20.0.0/24 side verify a client can still reach 10.20.0.10 and that the default route is intact - the fix adds a route, it does not touch the default.

3. A colleague proposes instead adding `ip route add 10.20.0.0/24 via <gateway>` on every client. Evaluate that option and state the condition under which you would accept it.

Read the worked answer

It treats a server-side defect as a client-side workaround: it requires touching an unbounded number of hosts, needs change approval in a domain you do not own, and leaves the server itself multi-homed with a wrong prefix - so server-initiated connections to 192.168.50.0/24 would still be broken, and any new client subnet would need the same patch.

It also risks masking the real fault: if rp_filter behaviour changes or the server's prefix is later fixed, the client routes become stale entries that can blackhole traffic in odd ways during failover.

Accept it only as a temporary measure if the server change cannot be scheduled inside the incident window, and then with conditions: a ticket to fix the server prefix, the client route added via configuration management rather than by hand, and a note that it must be removed after the server fix. Otherwise the /24 ip addr replace on the server is one command, reversible, and repairs both directions of traffic.