SD-WAN voice SLA collapse after adding LTE backup
RetailCo operates 300 stores. Each store has a FortiGate 81F with two 200 Mbps broadband links and an LTE backup, forming ADVPN spokes to two regional hubs. Voice runs on FortiFone over the corporate network; Microsoft 365 and Salesforce break out locally. The SD-WAN rule sends voice to ISP1 with a performance SLA of 120 ms latency, 1% packet loss, and 20 ms jitter; business traffic uses ISP1, general internet uses ISP2. After LTE was added as a third member, 40 stores report poor voice MOS during busy hours. IPsec tunnels to hub 1 flap about every 30 minutes, BGP sessions reset, and FortiManager shows duplicate performance-SLA probes sourced from the LAN and WAN interfaces. ISP1 latency is normally 35 ms but rises to 160 ms between 13:00 and 15:00; LTE latency is 90 ms but its monthly data cap is 20 GB. The business cannot accept voice outages but also cannot let LTE carry bulk traffic.
1. Diagnose why adding LTE degraded voice rather than improving resilience. Address SD-WAN member state, probe design, and asymmetric routing.
Read the worked answer
Why LTE degraded voice: LTE was added as an active SD-WAN member without a clear priority/cost bias, so voice could be steered to LTE or ECMP across members. LTE also changes the IPsec/BGP path: NAT on LTE, higher latency, and metered volume. Duplicate SLA probes suggest health checks are inherited by multiple rules or sourced from both LAN and WAN, so the SLA result is ambiguous. If BGP advertises the LTE prefix, return traffic can arrive on a different member than the outbound voice flow, causing asymmetry, reordering, and MOS collapse. Tunnel flapping every 30 minutes is consistent with DPD/BGP timers over a variable LTE path and with route churn when the SLA mark changes.
Diagnose: check diagnose sys sdwan health-check, diagnose sys sdwan member, diagnose netlink interface list, IPsec tunnel stats, BGP neighbor uptime, and FortiAnalyzer SD-WAN logs. Confirm which member carries voice during the 13:00-15:00 window and whether return traffic uses the same ISP.
2. Redesign the SD-WAN rules and health checks so voice fails over to LTE only when ISP1 truly breaches SLA, without consuming the LTE cap. Include specific thresholds and hold-down behavior.
Read the worked answer
Rule design:
- Voice: primary ISP1, backup LTE, priority 1 versus 10, SLA latency 120 ms, jitter 20 ms, loss 1%; hold-down 10-15 s, failback 60 s. Use a per-member health check with an explicit source interface and a stable probe target.
- Business apps: ISP1 primary, ISP2 backup, LTE never.
- Bulk internet: ISP2 only; add a lowest-priority rule or deny LTE for bulk traffic.
- Set LTE cost/priority high and exclude it from ADVPN shortcut unless ISP1 and ISP2 both fail.
Why: the goal is not 'use all links'; it is deterministic voice steering with LTE as last resort. A 20 GB cap can be consumed in hours by bulk traffic, so rule order and member priority must make LTE unattractive for anything except voice failover.
Thresholds: if ISP1 latency exceeds 120 ms for three consecutive probes, mark it down; require five good probes before failback. Use active probes carefully; active probes add little bandwidth. Monitor LTE byte counters and alert at 70% of cap.
3. What ADVPN and BGP changes would you make to stop the hub-1 tunnel flapping, and what would you monitor to prove the fix?
Read the worked answer
ADVPN/BGP changes: disable ADVPN shortcut for LTE unless required; set IPsec DPD to on-idle with a longer retry interval; use BGP hold/keepalive timers that tolerate LTE jitter (for example 30/90 s), and apply route-maps so hub-1 prefers ISP1 and does not install LTE routes unless ISP1 is down. Avoid advertising the LTE prefix into BGP by default. If flapping persists, use auto-discovery only on stable links and lower the LTE member priority.
Monitor: IPsec tunnel uptime per member, BGP neighbor transitions, SD-WAN SLA history, voice MOS by store, LTE data usage, and route-table changes. Prove the fix with a two-week baseline showing zero voice failovers to LTE during normal ISP1 conditions and no tunnel resets.