HTTPS Stalls and Large DNS Replies Fail Across an IPsec Tunnel
After a merger, head office reaches a new cloud VPC through a route-based IPsec tunnel whose interface MTU is 1400. Hosts on the office LAN use 1500. The cloud side blocks all ICMP except echo request and echo reply. Internal users report a clear pattern: the intranet portal's home page, which is small and plain HTTP, loads normally; the HTTPS login page starts to load and then hangs after the browser indicates it is negotiating; file downloads over the same HTTPS session stall at a few hundred kilobytes. Separately, the internal DNS server, queried across the tunnel, times out intermittently for names with large responses, while small responses resolve instantly. Packet captures on the office side show the client retransmitting identical segments of roughly 1400 bytes of payload with the Don't Fragment bit set, and never receiving an ICMP error or a smaller segment suggestion.
1. Explain the mechanism behind this pattern and why plain HTTP behaves differently from HTTPS.
Read the worked answer
This is a path MTU black hole. The client builds 1500-byte frames and sets Don't Fragment, as TCP does when performing path MTU discovery. The tunnel can carry only 1400, so the router in front of the tunnel must fragment or return an ICMP 'fragmentation needed' message. Fragmentation is not possible because DF is set; the ICMP message is dropped by the cloud-side policy that permits only echo. The client therefore never learns the real path MTU and keeps retransmitting the same oversized segment forever. Small segments pass, so the session is not dead - it is stalled.
Why the two protocols differ is a matter of message size. The plain HTTP home page is a few hundred bytes to a couple of kilobytes, which fits inside a single segment below 1400 bytes, so it transfers without ever probing the limit. The HTTPS login is where the TLS handshake sends the server certificate, typically 3–4 KB, in one or a few large records. Those segments exceed the path MTU, stall, and the handshake never completes. The file download stalls for the same reason once it tries to send full-size segments; the first few hundred kilobytes may get through in small pieces before the window fills with unacknowledged oversized data.
The DNS symptom has the same root: small UDP responses fit, large ones (DNSSEC, EDNS0 buffer sizes, many records) do not, and the TCP fallback that should rescue the query is blocked or also affected.
2. What would you change, and in what order, to fix this properly rather than patching the symptom?
Read the worked answer
- Let path MTU discovery work. Permit ICMP type 3 code 4 (fragmentation needed) inbound and outbound through the firewall policy. This is the correct long-term fix and it costs nothing; blocking it is the direct cause of the black hole. Also stop dropping IP fragments indiscriminately - if fragment handling is a security concern, use a policy that reassembles and inspects rather than discarding.
- Clamp TCP MSS at both tunnel endpoints. Set the adjustment so that SYN and SYN-ACK advertise an MSS consistent with a 1400-byte path, for example 1360 for IPv4 (1400 − 20 IP − 20 TCP). Clamping must be present on the router that sees the SYN, so apply it on both ends or you will fix one direction only.
- Make the tunnel MTU consistent on both endpoints and document it; a mismatch will resurface as an OSPF DBD problem or as one-way stalls.
- DNS specifics. Allow TCP 53 through the tunnel for fallback, and consider capping the EDNS0 buffer size on the resolver at 1232 bytes, which is the operational convention that avoids fragmentation on typical paths.
Order matters: allowing ICMP first restores self-healing behaviour for all protocols; MSS clamping then removes the dependence on ICMP for TCP; the DNS changes prevent the resolver from being the last service still broken.
3. How would you verify each fix, and what would you keep monitoring to catch a regression?
Read the worked answer
Verify at three levels: the packet, the connection, and the service.
- Path MTU check. Send a DF-set probe of a known size from a LAN host toward the cloud server, starting at 1400 and increasing until it fails. With ICMP permitted, you should see the fragmentation-needed message with the next-hop MTU, and the client should cache it. Repeat from the cloud side toward the LAN so both directions are proven.
- Handshake check. Capture the TLS handshake and confirm the certificate is delivered in segments no larger than the clamped MSS, and that no segment is retransmitted more than once. A clean handshake followed by a completed download of a file well above 1 MB is the practical proof.
- DNS check. Query a name known to return a large response and confirm it resolves within normal time over UDP, and that a forced TCP query also succeeds.
Monitoring for the next month: interface counters for fragments received and dropped, firewall hit counts on the ICMP rules (unexpected growth suggests a new MTU mismatch somewhere), retransmission rate and zero-window events on the tunnel, and resolver query timeouts. Add an alert on any firewall policy change that removes the ICMP type 3 code 4 allowance, because that single edit silently reintroduces the whole failure mode.