NCP-GENL: Jumbo Frames and a Silent Path MTU Black Hole

Jumbo Frames and a Silent Path MTU Black Hole

Two data centres are linked by a 10 GbE routed path. Storage hosts on VLAN 40 use jumbo frames: mtu 9000 on both NICs and on both access switches. The core router interface facing each site was rebuilt during a change window and left at the default mtu 1500; its firewall policy also drops inbound ICMP. Symptoms: ping with the default payload succeeds between sites, and a 4 KB scp works, but interactive SSH logins hang immediately after the banner and a 2 GB rsync stalls indefinitely at random offsets. On affected hosts ss -ti shows a send queue that never drains and repeated retransmissions. Application owners say the problem began right after the router change. The network team insists jumbo frames are configured correctly because the switch ports show MTU 9000. Traffic must stay on the routed path; no L2 stretch between sites is available, and the link is the only production replication path.

1. Explain the packet-level mechanism, including why small transfers succeed and SSH stalls after the banner.

Read the worked answer

This is a Path MTU Discovery black hole.

Storage hosts have mtu 9000, so TCP advertises MSS around 8960 and emits full-size segments with the DF bit set.
The router interface is 1500. It cannot forward a 9000-byte packet without fragmenting, and DF forbids that. Its correct response is ICMP type 3 code 4 ('fragmentation needed', next-hop MTU 1500). The firewall drops that ICMP, so the sender never learns and retransmits forever.
Anything that fits in 1500 bytes passes - hence default-payload ping and a 4 KB scp. SSH stalls right after the banner because the key-exchange packets that follow exceed 1500 bytes.
ss -ti with a send queue that never drains plus growing retrans is the signature: packets leave the host, are dropped in the path, and are never acknowledged, so the congestion window collapses and the transfer crawls or hangs.

The network team's evidence is a false negative. Switch ports at 9000 are necessary but not sufficient; jumbo frames need every L2 and L3 hop on the path, and PMTUD needs ICMP type 3 code 4 to be permitted end to end.

2. List the diagnostic steps and the specific evidence you would collect to prove the cause, and say why each step matters.

Read the worked answer

Establish the payload ceiling. ping -M do -s 8972 -c 3 <peer> (8972 + 28 = 9000). If that fails at 100% loss while -s 1472 succeeds, you have located an MTU cliff rather than a routing or firewall problem. Do it in both directions - asymmetric paths break in one direction only.
Find the narrow hop. tracepath -n <peer> reports the discovered MTU per hop and usually names the router interface directly; mtr --tcp --port 22 corroborates at the application level.
Prove ICMP is being dropped. On the sender, tcpdump -ni any 'icmp and icmp[icmptype]==3'. Silence is the evidence, not the absence of a problem. On the router, check ACL and firewall counters for denied ICMP.
Confirm interface values. ip link show on both hosts, the router's interface MTU, and the switch port MTU from show running-config interface. Build a table of every hop's MTU - the mismatch becomes obvious.
Confirm the application effect. ss -ti dst :22 showing non-zero retrans and an unacked, non-draining send queue, alongside a rsync that stalls at random offsets.

Record each result with timestamps so the change window can be tied to the failure without argument.

3. Recommend a remediation and rank the alternatives by blast radius. What do you change first, and how do you verify it?

Read the worked answer

1. Clamp MSS on the router (fastest, no host change). iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu rewrites the MSS in the SYN so neither side sends more than the path allows. Jumbo stays intact inside each data centre, the change is one rule, and it is trivially reversible. Cost: a little router CPU and an MSS value that no longer reflects the access link.

Also permanent, not optional: permit ICMP type 3 code 4 on the router's firewall. Without it, PMTUD stays broken for every protocol that sets DF, and the next MTU mismatch is silent again.

2. Drop VLAN 40 to mtu 1500 on the hosts. Simplest to reason about, but forfeits jumbo benefits (per-packet overhead and more interrupts at 10 GbE), and template drift will re-enable 9000 unless documented.

3. Raise the router and every transit interface to 9000. Best long-term if the media supports it, but it touches shared equipment, affects other VLANs still at 1500, and needs end-to-end validation before production.

Verification: ping -M do -s 8972 end to end in both directions, a 2 GB rsync with no stall, ss -ti showing stable retransmits, and a periodic check that records the effective path MTU so the next router replacement does not regress it.