SD-WAN Failover at a Distribution Centre: LTE Selected on a Healthy Fibre Link
A distribution centre runs a FortiGate 200F active-passive FGCP pair on FortiOS 7.4, with a 500/500 Mbps fibre circuit and a 100/20 Mbps LTE backup. Users include 600 handheld warehouse scanners talking to an on-premises warehouse management system and 120 office staff. Three weeks after migrating from static routes to SD-WAN, two symptoms appear. Outbound SaaS and scanner traffic intermittently egresses the LTE link while the fibre circuit is demonstrably healthy, and when fibre genuinely flaps, scanner sessions to the WMS reset and operators must re-authenticate. The design currently contains one policy route sending VoIP to the fibre next-hop, three SD-WAN rules with SLA targets, and a single health probe to 8.8.8.8. NAT is applied on both WAN interfaces, and the same public IP pair is used for inbound nothing.
1. Explain why LTE is being selected while fibre is healthy, and advise whether the VoIP policy route should stay.
Read the worked answer
Recall the FortiOS lookup order for a new session: policy route first, then SD-WAN rule, then the routing table. That single fact explains both halves of the question.
For the intermittent LTE selection, the SD-WAN rule is doing what it was told. If the fibre member is flagged out of SLA, the rule picks the next member, which is LTE. The likely cause is the probe itself: a single health check to 8.8.8.8 is easily rate-limited, ICMP-filtered or deprioritised upstream, so probes fail sporadically and fibre is declared dead while the circuit is fine. Confirm before changing anything by comparing the SD-WAN health-check output and link-monitor status against latency and loss measured from the fibre router, and look for a pattern aligned to probe loss rather than to user traffic. Fix by using several SLA probes including the next-hop gateway and FortiGuard targets, then tuning latency, jitter and packet-loss thresholds and failback timers so a single lost probe cannot trigger a move.
For VoIP, the policy route means those flows never consult SD-WAN health checks at all. There is no SLA-driven failover, and a fibre outage silently blackholes calls. Recommendation: delete the policy route and express VoIP as an SD-WAN rule with fibre as the preferred member and an appropriate SLA target, so it fails over on evidence rather than on hope. If the policy route must stay for a political reason, document that VoIP has no health-based failover.
2. When fibre genuinely flaps, why do scanner sessions to the WMS drop even though the pair is in HA? Recommend changes that reduce disruption.
Read the worked answer
HA protects the device, not the path. FGCP synchronises sessions to the standby unit, so the session table survives a chassis failure. What does not survive is the change of egress path: the public NAT address changes when traffic moves to LTE, the WMS replies to a source address that is no longer valid, and return traffic is dropped. If the LTE path re-enters the network elsewhere, asymmetric routing across the pair compounds it. Enabling source NAT re-evaluation on route change helps outbound sessions that are tolerant of a source-address change, and session pickup plus symmetric return checks remove some of the noise, but a TCP session bound to a dead NAT address will usually still break. Do not promise otherwise.
Design for application resilience instead:
- Give the WMS a stable address by carrying that traffic over an IPsec tunnel with consistent loopback-based source addressing, so the WMS peer sees one address regardless of the underlying WAN.
- Configure the SD-WAN rule with sensible failback and dampening so a one-second blip does not cause a move and a move back.
- Investigate the flap itself: fibre physical errors, upstream maintenance, or the health probe timeouts masquerading as a link failure.
- For scanners, confirm the application can resume after a brief loss, and treat the re-authentication requirement as an application issue rather than a firewall issue.
Finally, verify that the passive unit actually holds synchronised sessions, since a pair that has never failed is a pair whose failover is untested.