The 1 Gbps Backup Link That Delivers 6.5 Mbps
A branch office is 40 ms from the data centre, so round-trip time is 80 ms, and the path is provisioned at 1 Gbps with negligible loss. A single long-lived TCP connection copying backup data never exceeds about 6.5 Mbps, while four parallel streams reach 26 Mbps. Both endpoints run current Linux kernels and the socket buffers are set to 128 KB. The middlebox in the path is a stateful firewall configured to normalise TCP options, and it is known to drop option-kind bytes it does not recognise. The receive window observed in packet captures stays pinned at 65,535 bytes for the entire transfer, and there is no evidence of retransmission storms. The network team is asked whether the fix is more bandwidth, larger socket buffers alone, or something in the TCP handshake - and whether simply opening 160 parallel connections is an acceptable substitute.
1. Compute the achievable single-connection throughput and state whether this link is loss-limited or window-limited.
Read the worked answer
Throughput is capped by window divided by RTT: 65,535 B = 524,280 bits; 524,280 / 0.08 s = 6.55 Mbps. That matches the observed 6.5 Mbps almost exactly, which is the tell.
The pipe itself holds 1e9 × 0.08 = 80 Mbit = 10 MB in flight - about 153 maximal windows' worth. The sender would need to keep ~10 MB unacknowledged to fill it and is allowed ~64 KB.
Because the capture shows no retransmission storms and the path has negligible loss, congestion control is not the limiter: cwnd grows until the receiver's advertised window stops it. This is a receiver-window limitation, and roughly 99.3% of the provisioned capacity is idle. Adding bandwidth changes the BDP (making the ratio worse); it does nothing for the symptom. The observed 4× scaling with four streams confirms the diagnosis - each stream gets its own 64 KB, which is exactly the behaviour of a per-connection window cap rather than a shared bottleneck.
2. Do the window-scaling arithmetic: what shift is required, and is 160 parallel connections acceptable instead?
Read the worked answer
The window field is 16 bits, so the WS option multiplies it by 2^S, S ∈ [0,14]. Requirement: 65,535 × 2^S ≥ 10,485,760 B → 2^S ≥ 160.
- S = 7 → 8.4 MB → 8.4e6 × 8 / 0.08 ≈ 839 Mbps (short of line rate).
- S = 8 → 16.8 MB → enough, with headroom for RTT variation. So a shift of 8 is the minimum correct value here.
160 connections: arithmetically 160 × 65,535 ≈ 10.5 MB of aggregate window, and four connections already gave 4 × 6.5 = 26 Mbps, so it works. It is still the wrong answer: 160 slow-start ramps and 160 independent congestion controllers compete, cause loss-synchronised unfairness against every other flow on the path, multiply per-flow state in the normalising firewall, and cost 160 × 128 KB ≈ 20 MB of buffers per host. Treat it as a stop-gap for legacy appliances, never as the design.
3. The firewall drops unknown option bytes. What will the capture show, and what must be true end-to-end for the fix to work?
Read the worked answer
What you see. Window scaling is negotiated once, in the SYN and SYN-ACK, and cannot be renegotiated mid-connection. If the option (kind 3) is stripped from the SYN, both ends use a scale of 0 and the window field stays ≤ 65,535 for the life of the connection - precisely the pinned value in the capture. So the first diagnostic is to look at the handshake, not at steady-state packets.
What must hold end-to-end:
- The firewall must pass or ignore option kind 3 on SYN and SYN-ACK (or the tunnel/termination must sit above it). Some appliances also clamp the window, so verify the advertised scaled window in captures after the change.
- Both socket buffers must be ≥ BDP: 128 KB caps the window at 128 KB → ~13 Mbps even with scaling. Set tcp_rmem/tcp_wmem max to at least 12 MB.
- The receiving application must drain fast enough, or the advertised window closes regardless of buffer size.
- The sender must fill the pipe: slow start from ~10 MSS to 10 MB takes log2(10 MB / 1460) ≈ 13 RTTs ≈ 1.04 s per connection, so short transfers will never reach line rate - long-lived backup streams will.