Large Transfers Stall After 300 KB Across a Tunnel
Two offices are joined by a GRE-over-IPsec tunnel across an ISP that delivers a 1500-byte MTU on the customer handoff. The tunnel adds 24 bytes of GRE and roughly 44-60 bytes of ESP and outer IP, so the effective tunnel MTU is below 1500, but both tunnel interfaces are configured with mtu 1500 and no MSS clamping. Symptoms: SSH sessions connect and stay usable, ping with the default size works end-to-end, but scp of a 20 MB file transfers about 300 KB and then stalls until it times out; browser HTTPS uploads occasionally fail. ping -M do -s 1472 succeeds between the two LAN hosts but fails to the remote router loopback; ping -M do -s 1400 succeeds end-to-end. The edge firewall drops inbound ICMP type 3 code 4 as policy. Interface counters show no errors, but ss -ti on a stalled transfer shows retransmissions with an unchanged congestion window, and tcpdump on the sender shows the same sequence number retransmitted with DF set.
1. Diagnose the fault using the evidence, and explain why ping succeeds while scp stalls.
Read the worked answer
This is a path MTU discovery black hole. The tunnel cannot carry a 1500-byte inner packet, but the interface claims it can, so hosts build segments up to 1460 bytes of TCP payload and the encapsulated packet exceeds the ISP handoff. The DF bit is copied into the outer header, the packet is dropped, and the router that should signal the problem sends ICMP type 3 code 4, which the firewall discards. PMTUD therefore never learns the real limit, TCP retransmits the same segment with backoff, and the connection dies.
Every symptom fits. Default ping (56 bytes payload) never approaches the limit, and LAN-to-LAN ping at 1472 never enters the tunnel, so both succeed. The 1400-byte end-to-end success brackets the real MTU. Scp dies around 300 KB because the connection first completes the handshake with small packets, grows the window, then sends full-size segments that are silently dropped; SSH stays usable because interactive keystrokes are tiny. HTTP uploads fail only sometimes because most requests are small. The unchanged cwnd with repeated retransmission of one sequence number rules out congestion loss and confirms a persistent black hole.
Note that the sender cannot distinguish this from loss, so it keeps retrying into the same drop; counters show no errors because the drop is downstream.
2. Give the fix set, with values, ordered by where it belongs in the stack.
Read the worked answer
First, size the tunnel honestly. With 1500 bytes at the handoff, subtract 20 bytes outer IP, 8 bytes ESP header, 16 bytes IV, up to 15 bytes padding plus the pad-length byte, 12 bytes ICV, and 24 bytes GRE with key and sequence: the usable inner MTU lands near 1400. Set mtu 1400 on the GRE interface and on the IPsec policy so the stack fragments or signals correctly before encryption.
Second, clamp MSS at the tunnel boundary so end hosts never build oversized segments: iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1360 (1400 minus 20 bytes inner IP and 20 bytes TCP), or --clamp-mss-to-pmtu where both ends support it. Apply in both directions, and mirror the rule in the other site's firewall policy.
Third, allow ICMP type 3 code 4 through the edge policy. Fragmentation-needed is required for correct operation of any path whose MTU changes, and suppressing it breaks every protocol that relies on PMTUD, not just this tunnel. Rate-limit it rather than dropping it.
Optionally set the tunnel interface's MTU and let the kernel generate the ICMP itself. Then verify: iperf3 with an MSS of 1360 in both directions, sustained scp, and tcpdump confirming no DF drops at the handoff.
3. What are the failure modes of each candidate fix, and why is lowering the LAN MTU everywhere a poor choice?
Read the worked answer
Lowering the whole LAN to 1400 affects every host, including traffic that never enters the tunnel, and reduces throughput for local traffic by about 7% for no benefit. It also fails silently the next time the path changes, because the setting encodes a snapshot of the current encapsulation overhead rather than the actual path. And it still requires host-by-host reconfiguration of DHCP and static addresses; one forgotten server reintroduces the black hole.
MSS clamping has its own limits. It only rewrites TCP; UDP-based protocols such as DNS over 512 bytes, QUIC, IPsec transport peers and some backup tools will still emit oversized datagrams and still depend on ICMP. Clamping also fails if a middlebox strips the option or if the SYN is encrypted inside another tunnel. Allowing ICMP fixes the general case but exposes a rate-limited channel that some security policies dislike, and forged ICMP can force a low MTU, so rate-limit and validate source addresses.
Setting the tunnel MTU alone is insufficient if inner hosts never learn it, because PMTUD is exactly what is broken. The robust combination is a correct tunnel MTU, MSS clamping for TCP, and ICMP type 3 code 4 permitted, with monitoring that alerts when DF-dropped fragments appear at the handoff. Also document the encapsulation overhead so the next change does not repeat the mistake.