Large Responses That Never Arrive Across an IPsec Tunnel
A service in a data-centre region serves an API to a site reached over an inter-region IPsec tunnel with a configured tunnel MTU of 1400 bytes. Host NICs on both sides are at 9001 MTU on a jumbo-frame fabric. Clients at the remote site complete TCP handshakes and TLS handshakes normally, but the first response larger than about 1.4 kB hangs for roughly 30 seconds and then the connection resets. ss -ti on the server shows repeated retransmissions of the same segment with RTO backoff, and tcpdump shows the retransmitted segment carrying 1448 bytes of payload. Identical requests from clients in the same region as the server succeed immediately. The security group on the tunnel endpoints permits TCP 443 outbound and inbound, and drops everything else, including ICMP. Responses under 1.4 kB work from every location.
1. Distinguish between an MTU black hole and ordinary packet loss, using only the evidence given.
Read the worked answer
Packet loss and an MTU black hole both present as retransmissions, so the discriminator is which bytes fail and whether the failure is probabilistic.
Loss is probabilistic: retransmitted segments sometimes get through, recovery is eventual, and the pattern is not correlated with segment size. Here the failure is deterministic and size-correlated - small handshake segments and sub-1.4 kB responses always succeed, while the first full-size segment fails, and the same segment is retransmitted repeatedly and never arrives. The RTO sequence (roughly 0.2, 0.4, 0.8, … s, summing to about 30 s) matches the observed hang and reset.
The payload of 1448 bytes is the tell: that is 1500 − 20 (IP) − 32 (TCP with timestamps), i.e. the sender is still using a 1500-byte path MTU because path MTU discovery has not updated it. The tunnel can carry only 1400, so the encapsulated packet needs fragmentation, the DF bit prevents it, and the tunnel endpoint should return ICMP Type 3 Code 4 - but the security group drops all ICMP. The sender never learns, keeps retransmitting the same oversized segment, and the connection dies. That is a black hole by definition: the failure is silent because the notification is filtered.
2. What would you capture to confirm the diagnosis, and what result would refute it?
Read the worked answer
Capture at three points: server egress, tunnel ingress, tunnel egress. If the 1448-byte DF-set segment leaves the server, reaches the tunnel ingress, and does not appear at the tunnel egress, the tunnel is the drop point. Then inspect the tunnel interface counters for fragmentation needed/no route style drops and the endpoint's ICMP counters.
Confirm with a probe that bypasses TCP: ping -M do -s 1372 (1400 − 28) should succeed while -s 1472 fails, and tracepath should report the tunnel as the bottleneck MTU. On the wire, check that no ICMP Type 3 Code 4 ever reaches the server, and that the security group counters show the drops.
What would refute it: if the identical 1448-byte segment sometimes succeeds, or if small responses also stall occasionally, the size correlation is broken and loss, a stateful firewall timing out, or a middlebox is more likely. If the server does receive the ICMP frag-needed but ignores it, the fault moves to the host's PMTUD handling rather than the filtering rule. If captures show the packet never leaving the server, look at host MTU and routing first.
3. Recommend a fix set, and say which single change you would make first under change-control constraints.
Read the worked answer
The clean fix is to make the path MTU explicit rather than discovered through a blocked channel: clamp TCP MSS at the tunnel endpoints with iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu (or --set-mss 1360) so both directions negotiate a segment that fits 1400. This is a router-side change, needs no application rebuild, and is reversible.
Complementary measures: permit ICMP Type 3 Code 4 on the security group so PMTUD works for UDP and future paths; set the tunnel interface and host MTU for tunnel-destined traffic; and consider lowering the server's MTU on the tunnel-facing route if clamping is not possible on the provider side.
Under change control, do the MSS clamp first: it is the smallest, most targeted change, it fixes both directions, and it does not depend on an application release. Allow ICMP frag-needed second - it is a security-policy change and needs sign-off, and it restores correct behaviour for any protocol that cannot use MSS negotiation. Do not "fix" it by shrinking all payloads: that leaves every future protocol and every new path exposed to the same silent failure.