Single-Stream Replication Crawls on a 1 Gbps WAN Handoff
A candidate is paged about a nightly database replication between a Frankfurt data center and a Singapore warehouse. The provider delivers a 1 Gbps Layer 2 handoff, and ping between the two routers shows a steady 30 ms round trip with no loss. The job copies a 6 GB dump over one TCP connection from an rsync/SSH client. iperf3 -P 1 measures 18 Mbps; iperf3 -P 8 measures 760 Mbps. A capture on the Frankfurt router shows the sender's advertised window pinned at 65,535 bytes with repeated zero-window events. Separately, a new IPsec tunnel between two of the sites passes 200-byte pings but stalls whenever a file transfer starts; a 1,472-byte ping through the tunnel fails with 'fragmentation needed' while a 1,400-byte ping succeeds. The application team's proposal is to buy a second 1 Gbps circuit and bond it. Change control closes in 48 hours and the window is 30 minutes long.
1. Compute the bandwidth-delay product for the Frankfurt-Singapore path and explain why a single TCP stream stops near 18 Mbps. What single change yields the largest gain, and what must be verified at both endpoints?
Read the worked answer
BDP = bandwidth x RTT = 1,000,000,000 bit/s x 0.030 s = 30,000,000 bits, about 3.75 MB. That is how much data must be in flight to keep the pipe full.
A receiver advertising a 65,535-byte window without scaling caps one flow at 65,535 x 8 / 0.030 = about 17.5 Mbps, which matches the measured 18 Mbps. The path is not the problem: eight parallel streams reach 760 Mbps, so capacity and loss are fine. The limit is the receive window.
Largest single gain: enable TCP window scaling (RFC 7323). It is negotiated in the SYN/SYN-ACK options, so both stacks must offer it and no middlebox may strip it. Verify net.ipv4.tcp_window_scaling=1 on both hosts (default on modern Linux), check that receive buffer autotuning (net.ipv4.tcp_rmem) is not clamping the window, and confirm in the capture that the WSopt appears in both directions. After that, a single flow is also bounded by congestion-window growth, so expect a ramp rather than an instant jump; parallel streams or a WAN optimizer remain options for very high BDP paths.
Bonding a second circuit would not help this job: one flow still cannot fill one pipe, so the fix would add cost without changing the 18 Mbps ceiling.
2. The team also proposes enabling 9,000-byte jumbo frames end to end, including inside the IPsec tunnel. Evaluate that proposal against the actual bottleneck and the provider handoff.
Read the worked answer
Jumbo frames reduce packets per second and CPU/interrupt load, and they shrink per-packet header overhead, but they do not raise the window-imposed per-flow ceiling. At 30 ms RTT and a 64 KB window, you still get roughly 17.5 Mbps regardless of frame size. So the proposal does not address the measured bottleneck.
It also adds risk. Every Layer 2 and Layer 3 hop must agree on 9,000 bytes, including the provider's pseudowire; a single 1,500-byte hop silently black-holes large packets when DF is set. Inside the tunnel the arithmetic is worse: ESP adds roughly 50-73 bytes, so a 1,500-byte inner packet exceeds a 1,500-byte outer MTU. The correct tunnel treatment is ip mtu 1400 on the tunnel interface plus ip tcp adjust-mss 1360 on the tunnel and the LAN interfaces, so SYNs advertise an MSS the path can carry.
Recommended order: enable window scaling, clamp MSS, verify PMTUD with ping df-bit sweeps at 1,472 and 1,400 bytes, and only then evaluate jumbo frames on the LAN segments where all devices are controlled. Treat jumbo on the WAN as unsupported unless the provider states 9,000 bytes in writing.
3. The IPsec tunnel passes 200-byte pings but stalls on any file transfer. Explain the mechanism and give two concrete fixes with their trade-offs.
Read the worked answer
This is a PMTUD black hole. The inner packets are 1,500 bytes with DF set, ESP encapsulation pushes them over the path MTU, and the router that should return ICMP type 3 code 4 ('fragmentation needed') is either filtering ICMP or the message never reaches the sender. Small packets fit and pass, so the tunnel looks healthy until bulk data starts; the TCP session then retransmits, backs off, and appears to hang.
Fix 1, preferred: clamp MSS. Apply ip tcp adjust-mss 1360 on the tunnel interface (and on the LAN interface facing the servers) so TCP never sends a segment larger than the tunnel can carry. It is transparent to applications because it only rewrites the MSS option in SYNs, and it survives MTU changes.
Fix 2: set the tunnel MTU explicitly (ip mtu 1400) and permit ICMP unreachable messages through the ACLs so PMTUD can work; or, as a blunt instrument on older platforms, crypto ipsec df-bit clear lets the router fragment, which costs CPU and defeats PMTUD. Also test UDP-based applications separately: MSS clamping does nothing for them, so they rely on the tunnel MTU and correct ICMP handling.