Single-Stream Throughput and an MTU Black Hole on a 38 ms WAN

Bandwidth-delay product. At a 1 Gbps line rate with 38 ms RTT, the BDP is 1 × 10⁹ × 0.038 = 38 Mbit = 4.75 MB. That is how much data must be in flight to keep the pipe full. To merely reach the committed 400 Mbps you need 400 × 10⁶ × 0.038 / 8 = 1.9 MB in flight.

Back-calculate the observed rate. At 190 Mbps, the in-flight data implied by the same relationship is 190 × 10⁶ × 0.038 / 8 ≈ 0.9 MB. So the connection is behaviourally limited by roughly a 900 KB window, not by the link.

Why roughly 900 KB. With default Linux socket buffers and autotuning, the receive window grows only as the receiver demonstrates demand and the sender probes upward; a single stream over a long, thin path with any loss rarely ramps to the configured maximum. Window scaling must be negotiated at handshake time; if it is disabled on either end, the window is capped at 64 KB and throughput collapses to well under 20 Mbps, which the 190 Mbps figure rules out - so scaling is active but the buffer is filling to something near 1 MB.

Therefore: the 190 Mbps is a sender/receiver window limit, not a carrier fault. The eight-stream test reaching 900 Mbps confirms the path itself is healthy, because eight streams each with a smaller window add up. This is also why an SLA claim would fail: the five-minute average may well be under 400 Mbps, but the cause is the application.

Is 16 MB the right number? 16 MB would support 16 × 10⁶ × 8 / 0.038 ≈ 3.4 Gbps, far beyond the link. About 8 MB per socket is sufficient; the rest is bufferbloat, memory footprint and latency under contention. With 16 MB × 20 concurrent streams × 2 directions, you would reserve hundreds of MB for no benefit.

The symptom pattern is a path-MTU black hole. Small packets (ping, SSH banners) succeed because they fit inside the tunnel MTU. Large transfers complete the handshake and then stall: full-size segments carry the Don't-Fragment bit, exceed the tunnel's usable MTU, and are dropped somewhere in the path. The ICMP 'fragmentation needed' message that would tell the sender is filtered by the site firewall, so the sender never learns and keeps retransmitting the same oversized segment. Classic signature: connection establishes, throughput goes to zero or to a trickle, no useful error appears on either host.

Arithmetic. Start at 1500 bytes. ESP in tunnel mode with AES-GCM consumes an outer IP header (20 bytes), an ESP header and IV (about 16 bytes), and the ESP trailer plus ICV (about 16–32 bytes, more with padding). That is roughly 60–80 bytes, leaving an inner MTU of about 1400–1420 bytes, and an inner TCP MSS of 1400 − 40 = 1360 bytes.

Fixes, in order of preference:

1. MSS clamping on the tunnel endpoints (ip tcp adjust-mss 1360 or the equivalent for the platform), which is stateless, needs no ICMP and is robust to the firewall policy.
2. Set the tunnel interface MTU explicitly to the calculated value and ensure the inner hosts derive MSS from it (PMTUD on the tunnel path).
3. Permit ICMP type 3 code 4 through the firewall as a general policy, because other paths will hit the same problem.

Verification: transfer a file larger than 10 MB through the tunnel with ping -M do -s 1400 probes to find the real maximum, and confirm with a packet capture that no segment exceeds the tunnel MTU. A WAN accelerator would not fix this; it would sit behind the same black hole.

Sequence by cost and by what each step proves.

1. Fix the MTU black hole first (MSS clamping, tunnel MTU, ICMP policy). It is a configuration change, it is currently breaking functionality rather than merely slowing it, and it costs nothing. Do not bundle it with performance tuning, because you want to know which change fixed which symptom.
2. Tune the single stream. Confirm window scaling is negotiated, raise tcp_rmem/tcp_wmem maxima to about 8 MB, and enable pacing or BBR-style congestion control if the kernel supports it. Re-measure. Expect 600–900 Mbps if the path is genuinely clean.
3. Parallelise the transfer. Run four to eight streams, or use a transfer tool designed for high-BDP paths, and schedule the nightly job outside peak hours. This is the cheapest way to reach line rate and it also reduces sensitivity to single-flow loss.
4. Only then consider a WAN accelerator. Justify it only if steps 1–3 leave the nightly window too short, and price it against the file volume: at 900 Mbps versus 190 Mbps, a 60 GB file takes about 9 minutes instead of 42, so the business case is about the batch window, not about the bandwidth itself.

On the SLA claim: do not file one yet. The committed rate is 400 Mbps averaged over five minutes, and the single-stream result is an end-host limitation; the eight-stream test at 900 Mbps is evidence the carrier is delivering. Filing a claim you cannot substantiate damages the relationship and wastes the credit. If, after steps 1–3, a multi-stream test still averages below 400 Mbps over five minutes during the transfer window, then you have the evidence - and the measurements to attach to the ticket.