ERP Asymmetry and BGP Flapping After Adding 5G
Helios Components runs three plants on FortiGate 100F firewalls. Plant A has an MPLS circuit with eBGP to the carrier, a 500/500 Mbps DIA circuit with a static default, and a 5G backup. Plants B and C are similar. SD-WAN rules send ERP traffic to the data center over MPLS, Microsoft 365 over DIA, and everything else load-balanced. After 5G was added as an SD-WAN member, the carrier's eBGP session over MPLS flaps every 30-40 minutes. When MPLS latency exceeds the SLA, the ERP rule fails over to DIA, but users report one-way audio and dropped sessions. The carrier says the plant's traffic arrives on MPLS but returns on DIA. The plant needs ERP to stay stable during MPLS micro-outages without giving up 5G resilience.
1. Why would adding 5G cause the eBGP session to flap? What would you check first?
Read the worked answer
The most likely cause is that the BGP TCP session itself is being steered by SD-WAN. SD-WAN service rules are evaluated top-down; a broad rule or the implicit rule can match TCP/179 to the carrier peer and send it out 5G. The carrier then sees the BGP session from a different source address and resets it. Check diagnose sys sdwan service for a service matching the peer, get router info bgp neighbors for the last reset and source address, and diagnose sys sdwan health-check status for 5G. The fix is a top rule that matches the BGP peer address and TCP/179 and pins it to MPLS only, plus a BGP update-source on the MPLS interface. Do not rely on the implicit rule for control-plane traffic.
2. Why does ERP fail over to DIA but still break, and how do you make it symmetric?
Read the worked answer
SD-WAN only chooses the egress path at the plant. The return path is chosen by BGP at the data center. If the data center still prefers MPLS, an ERP session that egresses DIA will return over MPLS and be dropped by stateful inspection or arrive with the wrong source. The fix is either to keep ERP on MPLS and use 5G as the only backup, with the data center also having a route over 5G and matching local preference, or to make DIA a real BGP path in the same design so both directions move together. A common safe design is MPLS primary and 5G secondary for ERP, with DIA excluded from that rule; use DIA for Microsoft 365 and general internet. Test with a traceroute in both directions.
3. Give the SD-WAN rule order and health-check design you would use.
Read the worked answer
Top: BGP peer, MPLS only. Next: ERP, match the ERP server or internet service, use SLA mode, members MPLS then 5G, with an SLA for latency and jitter. Target the ERP server or carrier PE, not a public DNS. Next: Microsoft 365 internet service over DIA then 5G, load-balanced if both pass. Last: default load-balance DIA and 5G with MPLS as a low-priority member. Health checks should use the same probe type the application uses, for example TCP 443 to the ERP server, and set the source per member. Confirm with diagnose sys sdwan service and with a forced MPLS degradation test before changing production.