F5-CSE: Branch users authenticate but stall on any response larger than 1.3 KB

Branch users authenticate but stall on any response larger than 1.3 KB

A web portal is published on a BIG-IP running TMOS 16.1 at 203.0.113.10:443, using a standard virtual server, a client SSL profile, and a pool of three members on the internal VLAN. Head-office users work normally. Branch users reach the portal through a site-to-site IPsec tunnel whose effective MTU is 1400 bytes. They load the login page and authenticate successfully, but any request that returns more than about 1.3 KB hangs until the browser times out; a manual retry sometimes succeeds. A capture on the BIG-IP external VLAN shows the BIG-IP sending full 1500-byte segments toward the branch, retransmitting each one five or six times with growing intervals, and never receiving an ACK for those bytes. No ICMP 'fragmentation needed' messages appear in the capture. Responses under about 1300 bytes are delivered correctly. The branch edge firewall drops inbound ICMP type 3 code 4 by policy. The external VLAN is still set to the default 1500-byte MTU.

1. Explain the failure mechanism and why authentication succeeds while the pages behind it hang. Which element in the path is actually at fault?

Read the worked answer

The BIG-IP terminates the client-side TCP connection, so it builds its own segments using the MTU of the egress VLAN. That VLAN is at 1500, so the BIG-IP advertises MSS 1460 and sends payload-carrying segments of up to 1500 bytes, with the DF bit set.

Those packets cannot cross a 1400-byte tunnel intact. The correct behaviour is for the tunnel head-end to return ICMP type 3 code 4 (fragmentation needed, DF set) so the sender reduces its segment size (path MTU discovery). That ICMP is dropped by policy, so the BIG-IP never learns the real limit and keeps retransmitting the same oversized segment until the connection dies. This is a classic PMTUD black hole.

Why authentication works: the login page, its TLS records and its HTTP responses all fit inside a single segment under 1400 bytes, so nothing ever needs a full-size packet. The failure only appears once a response needs one.

Who is at fault: not the pool or the health monitors. Members are up, the server answered, and the capture shows the problem purely in the BIG-IP to client direction. Two parties share the blame: the network path drops the ICMP signal that would let the BIG-IP adapt, and the BIG-IP is configured to assume a 1500-byte path to a branch that does not have one.

2. What changes would you make, on the BIG-IP and elsewhere, and what does each option cost you?

Read the worked answer

Option A - shrink the BIG-IP side of the path (fast, self-contained). Lower the external VLAN MTU to 1400 so the BIG-IP advertises MSS 1360 and never emits 1500-byte packets toward the branch. Command: tmsh modify net vlan external mtu 1400, then tmsh save sys config. Cost: it applies to every virtual server and every client on that VLAN, including head-office users on a clean 1500-byte path, who lose a little throughput to the smaller segment size. It also only fixes the BIG-IP to client direction; client to BIG-IP uploads still rely on the branch side doing the right thing.

Option B - fix the path (correct long term). Allow ICMP type 3 code 4 back to the BIG-IP at the branch firewall, or clamp MSS to 1360 on the tunnel head-end so neither end ever emits an oversized frame. Cost: cross-team change, firewall policy exception, and a review of whether other ICMP types are also being dropped.

Option C - scope it. Put branch traffic on its own VLAN or virtual server with the smaller MTU, leaving everyone else at 1500. Cost: more objects and routing changes.

A sane sequence is A now to stop the bleeding, then B, then revert A if you want the whole estate back on 1500. Do not simply disable PMTUD or clear the DF bit as a permanent fix; you are hiding the signal, not removing the constraint.

3. How would you prove the diagnosis before and after the change, and what other faults could produce the same symptom?

Read the worked answer

Before the change:

From a branch host: ping -M do -s 1472 203.0.113.10 fails, while -s 1372 (1372 + 28 = 1400) succeeds. That isolates the MTU as the boundary.
On the BIG-IP: tcpdump -ni external -vv host <branch-ip> and read the MSS option in the SYN-ACK (1460 today, 1360 after) and confirm DF is set on the retransmitted full-size segments.
Run curl -o /dev/null against a large object from the branch and time it, so you have a measurable before/after.

After: re-run the same three checks. The SYN-ACK MSS should drop to 1360 and the large download should complete.

Differential diagnosis - what else looks similar:

A proxy or firewall that silently drops fragments rather than signalling: you would see the same retransmission pattern but ping with -M do would succeed at every size.
A pool member on a smaller-MTU segment: retransmissions would be in the member-to-BIG-IP direction, not the BIG-IP-to-client direction.
A zero-window condition or a stalled application: the capture shows window updates or server think-time, not repeated full-size data segments with no ACK.
TLS renegotiation or a broken server certificate chain: the handshake fails, rather than completing and then stalling mid-body. The discriminator here is retransmission of full-size data segments with no ACK and no ICMP feedback.