Silent Packet Loss in a Dual-Underlay Overlay
Users at a branch report intermittent application timeouts (5–10 seconds) every few minutes. The branch has fiber and LTE underlays in an active/active overlay with per-packet load balancing enabled. BFD is configured with 300ms × 3 on both tunnels. The branch edge shows tunnels up, low average latency (22ms), and no BFD flaps in the last 24 hours. However, interface counters show LTE TX errors climbing and the fiber interface shows occasional output drops during microbursts. Application flows are TCP and are reordering-sensitive.
1. Given the symptoms, identify the most likely root cause and explain why BFD and average latency did not catch it.
Read the worked answer
Most likely root cause: per-packet load balancing across paths with different latency and loss characteristics, combined with LTE TX errors and fiber microburst drops.
Why BFD missed it:
- BFD detects path down, not path degraded. LTE with TX errors still passes BFD probes if enough get through; 300ms × 3 may not trip on intermittent loss.
- BFD probes are small and may not experience the same microburst drops as bursty application traffic.
Why average latency missed it:
- Averages hide tail events. A 5–10s timeout every few minutes is a tail phenomenon; mean latency of 22ms is consistent with mostly-good path plus occasional multi-second stalls.
- Per-packet load balancing across LTE and fiber causes TCP reordering, which triggers duplicate ACKs, spurious retransmits, and congestion-window collapse - perceived as timeouts.
The LTE TX errors and fiber output drops are the packet-loss sources; per-packet LB amplifies them into TCP-level stalls.
2. What configuration changes would you make, and in what order, to resolve the issue with minimal risk?
Read the worked answer
Order of changes (least risky first):
- Disable per-packet load balancing; switch to per-flow hashing. This alone often eliminates the reordering-induced timeouts. Low risk, immediate.
- Investigate LTE TX errors - physical layer (signal, cabling, carrier). If LTE is unreliable, demote it to backup for reordering-sensitive apps rather than active/active.
- Address fiber microbursts - check interface speed/duplex, enable egress shaping to smooth bursts, review QoS queues for drops.
- Tune BFD to be more sensitive to loss (e.g., add loss-based SLA probes) - but do not rely on BFD alone for degradation.
- Add AAR with p99 loss/latency/jitter SLAs so degraded paths are avoided proactively, not just when BFD fails.
- Application-aware steering: pin TCP/reordering-sensitive apps to fiber; use LTE only for failover or for UDP/latency-tolerant traffic.
Validate each step with before/after measurements of timeouts and TCP retransmits.
3. How would you prove the fix worked and distinguish it from a coincidental improvement?
Read the worked answer
- Baseline first: capture timeout count, TCP retransmit rate, and p99 latency per path for 24h before changes.
- Change one variable at a time (per-flow LB first) and measure for a full business cycle.
- Compare distributions, not means: p99/p999 latency, retransmit %, and timeout events per hour.
- Correlate with path: confirm timeouts dropped on fiber-pinned flows and that LTE TX errors no longer coincide with stalls.
- Negative control: if possible, leave one site unchanged and confirm it still shows the symptom - rules out a carrier-side or application-side coincidental fix.
- Sustained observation for at least a week, since the symptom is intermittent.