SMB, HTTPS, and RDP Stall Across an IPsec Tunnel
Silver Creek's plant connects to HQ through a site-to-site IPsec tunnel on ISR 1111 routers. The tunnel interface MTU works out to 1400 bytes; both LANs are 1500. Three weeks ago the plant firewall's policy was refreshed, and since then: SSH from HQ to plant servers works, ping with 1000-byte payloads works, but SMB copies stall at about 1.2 MB, HTTPS to the plant's internal application hangs during or immediately after the TLS handshake, and RDP sessions freeze within about 20 seconds. Workstations run Windows with PMTUD enabled. The firewall's outside ACL now denies all ICMP, including type 3 code 4 and type 11, which the previous policy permitted. From an HQ workstation, ping -f -l 1380 to a plant server gets no reply; ping -f -l 1350 gets replies. show crypto ipsec sa on both routers shows encapsulations and decapsulations incrementing in step with no drops recorded. The plant team says HQ 'is blocking us'; the HQ firewall team replies that the tunnel is healthy, which the counters support.
1. Interpret the ping results and explain why SSH succeeds while SMB, HTTPS, and RDP fail. Give the MTU arithmetic explicitly.
Read the worked answer
The ping ladder
ping -f -l 1350succeeds: 1350 payload + 8 ICMP + 20 IP = 1378 bytes on the wire, comfortably under the 1400-byte tunnel MTU.ping -f -l 1380fails: 1380 + 28 = 1408, just over 1400. With DF set, the router cannot fragment and must return ICMP type 3 code 4 - which the new ACL drops. So the sender sees silence, not an error.
That bracketing is a normal PMTUD probe. The path MTU is somewhere between 1378 and 1408, and we already know the tunnel MTU is 1400.
Why the split in application behaviour
- SSH works: interactive keystrokes and small responses fit in one or two small segments. Nothing ever needs a 1400+ byte packet.
- SMB stalls at ~1.2 MB: the copy succeeds until a full-size 1460-byte TCP segment is retransmitted forever. A few hundred KB of buffered writes is exactly the pattern of 'copy starts, then hangs'.
- HTTPS hangs at handshake: a server certificate chain routinely exceeds 1400 bytes, so the handshake cannot complete.
- RDP freezes: session setup exchanges and screen updates produce large frames; the session dies when the first oversized segment is black-holed.
The tunnel is not dropping packets - it is doing its job and generating the ICMP unreachable that the firewall now discards. That is the definition of a PMTUD black hole.
2. Who owns the fault: the tunnel, the firewall ACL, or the servers? Justify the attribution and say why 'the tunnel is healthy' is both true and irrelevant.
Read the worked answer
Verdict: the firewall ACL change is the root cause.
The 1400-byte tunnel MTU is expected and correct. Encapsulation overhead is a property of IPsec, not a fault, and it worked fine for years because PMTUD was allowed to function. The change inside the last three weeks is the ACL refresh that now denies ICMP type 3 code 4. That single edit converts a normally self-healing condition into a permanent black hole.
Why 'the tunnel is healthy' is true but irrelevant
The HQ firewall team is reading show crypto ipsec sa and seeing encaps/decaps incrementing evenly with no drops - that only proves the tunnel is passing packets. It says nothing about whether the path can discover its MTU. The tunnel is the party that generates the ICMP unreachable; the ACL on the far side is the party that eats it. Both statements can be true simultaneously, and the one that identifies the fault is the ACL.
Servers are not at fault. They are behaving per RFC: setting DF and relying on ICMP feedback. Windows' PMTUD black-hole detection will sometimes recover by shrinking MSS after repeated loss, which is why the symptom is 'slow and flaky' rather than totally dead - but it is not something to depend on.
The correct blame statement for the ticket: a firewall policy change removed the one signal the path needs. Everything else is a contributing weakness.
3. State the fixes - including router configuration - and explain which symptom MSS clamping will not solve.
Read the worked answer
Primary fix: restore the ICMP signal. Permit ICMP type 3 code 4 (fragmentation needed, DF set) and type 11 code 0 from the tunnel endpoints and from plant-side routers, at minimum toward HQ subnets. This repairs PMTUD for every protocol at once and is the correct, permanent answer. Blanket ICMP denial is a recurring cause of exactly this class of 'random app hangs'.
Secondary defence: clamp MSS on both routers.
interface Tunnel0
ip tcp adjust-mss 1360
ip mtu 1400
1360 is chosen so that 1360 + 20 TCP + 20 IP = 1400 fits under the tunnel MTU with the IPsec header accounted for by the ip mtu setting. Applying it on both ends covers traffic in both directions. The clamping rewrites the MSS option in the SYN, so TCP endpoints never offer a segment that will not fit.
What MSS clamping will not fix:
- Non-TCP traffic - large UDP datagrams (big DNS responses, TFTP, some RDP UDP transports, SNMP bulk) still exceed the path MTU and still depend on ICMP.
- Traffic between hosts that never crosses the clamped interface (for example plant-to-plant on the same LAN, or anything the tunnel router is not forwarding).
- Diagnostics:
ping -fwill still fail, so troubleshooting stays confusing for the next engineer.
Treat clamping as insurance, not as a substitute for unblocking the ICMP the network legitimately needs.