The 200 Mbit/s WAN That Feels Like Dial-Up
A logistics firm centralised its ERP client in a Dallas data centre serving six depots, including Singapore and Perth. Last quarter it upgraded every WAN circuit from 20 Mbit/s to 200 Mbit/s MPLS at an extra USD 18,000 per month. Invoice posting still takes about nine seconds per transaction - the same as before. Round-trip latency Dallas–Singapore is 380 ms with 3 ms jitter and no measured loss, and bulk file transfers between servers at those two sites reach 150 Mbit/s. The ERP client opens three TCP connections and performs roughly 14 sequential request/response round trips per posting, moving about 120 KB per transaction. The application vendor's written recommendation is to add more bandwidth. The CFO wants a technical explanation before approving any further spend.
1. Using the numbers given, explain why the upgrade did not improve posting time, and quantify what would.
Read the worked answer
The workload is latency-bound, not bandwidth-bound.
- Bandwidth-delay product: 200 Mbit/s x 0.38 s = ~9.5 MB of data would have to be in flight to fill the pipe. One posting moves 120 KB - about 1% of that.
- Per-connection ceiling from the receive window: 64 KB / 0.38 s = ~1.35 Mbit/s per connection, ~4 Mbit/s across the three connections. The 200 Mbit/s circuit is irrelevant.
- Round-trip floor: 14 round trips x 380 ms = 5.3 s of pure waiting, before server processing, queueing and client render time. Add ~2–3 s of server work and you get the observed 9 s.
The 150 Mbit/s bulk transfer confirms the circuit and its window scaling are healthy; nothing here points at the carrier.
What would actually move the number: cutting round trips (batch the 14 calls, reuse a pooled connection, cache reference data locally) or cutting distance (app mirror or VDI near the users). Each round trip removed saves 380 ms; removing 8 of the 14 saves ~3 s. Buying more bandwidth saves zero.
2. What would you measure in the next 48 hours, and how do you separate latency from server-side delay from packet loss?
Read the worked answer
Capture at the client NIC and timestamp against an NTP-synced clock. For each of the 14 exchanges, split the timeline into three buckets: client-to-server propagation, server response time (first byte after request arrives), and client think/render time.
- Latency check: sum of the 14 round trips should land near 5.3 s. If it does, latency dominates and the arithmetic above holds.
- Loss check: look for retransmissions and out-of-order segments in the capture. A single lost segment in a 120 KB transfer can add a full RTO (often 200 ms+, doubling on repeats). Zero loss in ICMP does not rule out microburst loss; the TCP trace does.
- Server check: compare wire timings to application APM or server-side logs. If the server takes 250 ms per call, 14 calls is 3.5 s of the 9 s and the network story is only half the answer.
Also probe the path in both directions from each depot - asymmetric routing or a congested return path shows up here and nowhere else.
3. Rank the remedies by cost and benefit, and state which options you would reject.
Read the worked answer
Ranked:
- Application change - batch or eliminate round trips, connection pooling, local caching. Near-zero capex, removes the dominant term. Requires vendor engagement; highest benefit per dollar.
- Move compute closer - regional app mirror or VDI broker per region. Moderate capex, cuts RTT to ~10 ms, but adds a patching and licensing footprint.
- WAN optimisation / TCP proxy. Helps bulk and CIFS, gives little for a 14-round-trip transactional app, and breaks TLS inspection unless certificates are re-signed. Treat as a partial fix.
- Lower-latency path (dedicated cloud/colocation interconnect). Effect is real but lead times are months and cost is high; only worth it if the app cannot change.
Reject: another bandwidth upgrade. The BDP and the 150 Mbit/s bulk test prove the pipe is not the constraint, so the spend cannot improve the metric the CFO is asking about. Also reject 'increase the ERP client timeout' - it hides the symptom and creates silent failures.