I85PREP
⌘ K
NetSec Professional study materials
NetSec Professional study resources

The 4% HTTPS Stall After Enabling ECMP

NetSec Professional · Put your knowledge to work in context.

All case studies

The 4% HTTPS Stall After Enabling ECMP

A stateful firewall at the HQ edge has two internet uplinks: ISP-A fiber at 500/100 Mbps with a 1500-byte MTU, and ISP-B fixed wireless at 200/50 Mbps delivered over PPPoE with a 1492-byte MTU, where the carrier's tunnel further reduces usable MTU to 1400 bytes. Last week you replaced static default routes with ECMP across both uplinks using 5-tuple hashing. Since then about 4% of outbound HTTPS sessions stall for 20-30 seconds, then succeed after the browser opens a new connection. ICMP pings to the same destinations always work, and the problem never occurs for sessions leaving via ISP-A. Interface counters show no errors, CPU is at 35%, and the session table is at 60% of capacity.

1. What is the most likely cause, and what evidence would you collect to confirm it before changing any configuration?

Read the worked answer

Most likely cause: an MTU black hole on the ISP-B path. The TCP handshake completes because SYN/SYN-ACK are small, but the server's first large segment (a TLS ServerHello carrying the certificate chain, typically 1400-1500 bytes) is sent with DF set, exceeds the 1400-byte path MTU, and the required ICMP Fragmentation Needed is filtered somewhere in the carrier path. The client retransmits, the firewall session persists, and after 20-30 s the browser gives up and opens a new connection, which may hash to ISP-A and work. That also explains why a small, round-robin share of flows (roughly proportional to the ISP-B hash share) is affected and why ping always succeeds.

Evidence to collect before changing anything:

  • Confirm the effective MTU per path with DF-set probes of decreasing size (e.g., ping with 1472/1400/1372 payload and DF) out of each interface, sourced from the firewall.
  • Run a packet capture on the ISP-B interface for one failing flow: look for repeated retransmissions of one large segment with no ACK, while the handshake is already complete.
  • Check whether the firewall or upstream is dropping ICMP type 3 code 4 (counters, or capture inbound on ISP-B).
  • Correlate session logs: group failing sessions by egress interface to verify the failure set is essentially all ISP-B.
  • Verify no MSS clamping is currently applied on either interface.

Do not accept 'ISP-B is flaky' as a diagnosis; the deterministic size dependence and clean counters point away from loss.

2. What configuration changes would you apply, and what side effects or limitations should you warn about?

Read the worked answer

Fix on the ISP-B path only, and keep ISP-A untouched:

  1. Apply TCP MSS clamping on ISP-B egress and ingress for forwarded traffic: MSS = 1400 - 40 = 1360 bytes. This is the least invasive fix because it affects only TCP traffic on the constrained path and requires no change to ISP-A.
  2. Alternatively (or additionally, as a backstop) set the ISP-B interface MTU to 1400 so the firewall fragments or refuses oversize packets locally rather than black-holing them.
  3. Make sure ICMP Fragmentation Needed is permitted inbound on ISP-B if policy allows, so path MTU discovery can work for non-TCP protocols.

Side effects and limits to state explicitly:

  • MSS clamping is TCP-only. Large UDP with DF set (some QUIC, IPsec, DNS-over-HTTPS fallback, SIP) can still black-hole; QUIC in particular may need to be blocked or forced to TCP fallback if it misbehaves.
  • Clamping slightly increases packet count and header overhead; it does not reduce throughput materially at 200 Mbps.
  • Clamping globally on both interfaces 'to be safe' hides the real problem and penalizes the healthy 1500-byte path.
  • Verify with a capture that SYN options now advertise 1360 on ISP-B, and re-test a known failing HTTPS destination.

3. Longer term, would you keep ECMP? Describe the routing/forwarding design you would choose and how you would monitor it.

Read the worked answer

ECMP with 5-tuple hashing is a poor fit here because the two paths differ in capacity, latency, MTU, and reliability, yet ECMP treats them as equal. A better design:

  • Use policy-based/SD-WAN path selection rather than pure ECMP: ISP-A preferred for latency-sensitive and interactive traffic, ISP-B used for bulk or as failover, with health probes (HTTP/DNS, not just ICMP) per path.
  • Keep per-flow stickiness (hash on the full 5-tuple, never per-packet) so a flow does not move mid-session; if you use per-interface SNAT, source translation must follow the chosen egress interface so replies return on the same path and do not trip reverse-path checks.
  • Apply the MTU/MSS fix independently of the path-selection decision, since failover to ISP-B would otherwise reintroduce the stall at the worst possible moment.
  • For failover, size timers so a dead ISP-A is detected in seconds, and ensure existing sessions are expected to reset rather than silently black-hole.

Monitoring: per-path probe latency/loss/jitter, per-interface session counts and SNAT port utilization, MSS-clamp counters, and an alert on retransmission-heavy flows per egress interface. Sample a weekly synthetic HTTPS fetch forced out each path at 1400 and 1500 bytes to catch regressions.