HQ HA Pair With SD-WAN Health-Check Flapping

The GUEST VDOM being unaffected is the first clue: a chassis-level HA event would not spare one VDOM while breaking another, especially when GUEST egresses through the untouched ISP-B path. The absence of an HA transition in the logs points the same way.

Evidence to collect:

• diagnose sys ha status and get system performance status: confirm the primary has been up for weeks and the secondary has not taken over. Check HA logs for state transitions, not just syslog alerts.
• diagnose sys sdwan health-check status: inspect per-member latency, jitter, loss and current state during the 10:00 window. This is the decisive output.
• Event-log entries for SD-WAN health-check or member status changes, correlated to the exact minute sessions dropped.
• get router info routing-table all before, during and after: confirm whether the default or SD-WAN routes for ISP-A are withdrawn and reinstalled when the member flips.
• diagnose sys session list filtered on the affected server port: check whether sessions show a change of source NAT address or egress interface.
• For the tunnel, diagnose vpn ike gateway list and tunnel statistics: confirm whether the tunnel was torn down because the underlying route disappeared rather than because of IKE failure.

If HA status is stable and the health-check state flips at the same timestamps as the user complaints, the cause is an SD-WAN path transition, not a failover. That distinction matters because the remediation is completely different: HA tuning would do nothing here.

Session pickup replicates sessions to the secondary unit so they survive an HA failover. It does nothing when there is no failover. Here the firewall stays primary, but the SD-WAN member carrying the traffic is declared down, so the route toward ISP-A is withdrawn and traffic is re-evaluated toward ISP-B.

The sessions break because NAT is bound to the old path. Outbound sessions were translated to the ISP-A interface address, or to an IP pool tied to ISP-A. When the member is marked down, return traffic from the internet still targets the ISP-A address, which is no longer reachable or no longer answered by the firewall. Packets that do arrive on ISP-B fail the stateful or reverse-path check, so clients see retransmissions followed by resets or timeouts. The branch tunnel drops for the same reason: its outer packets lose the underlying route and IKE tears the tunnel down, then rebuilds it.

Practical mitigations, in order of value:

• Make the member-down decision less trigger-happy. Raise the health-check fail time and recovery time and set SLA thresholds that LTE can realistically meet. A saturated fibre link is degraded, not dead.
• For business-critical traffic, use a manual or priority-based SD-WAN rule so ISP-B is used only on a real link failure, not on an SLA violation.
• Give the branch redundant tunnels, one per member, with BGP so it reconverges without waiting for a full DPD-driven teardown.
• Keep NAT pools per member and never mix addresses across members; document that any path change will still break existing sessions that are NATed through the old member.

Accept that some breakage is inherent: you cannot move a live NATed flow to a different public address without breaking it.

Sequence:

1. Baseline. Export the config through FortiManager and record the current revision. Capture a week of health-check statistics so the new thresholds are based on measured loss and jitter, not guesswork.
2. Schedule a window outside 10:00 and outside the backup job, with the branch owner and helpdesk informed. Notify users that new sessions may briefly re-establish.
3. Change health-check settings first: interval, fail time, recovery time, and SLA targets that reflect what LTE actually delivers. Use FortiManager install preview to confirm the diff touches only the intended objects.
4. Add the critical-application SD-WAN rule with a manual or priority strategy, so only a genuine link-down event moves that traffic.
5. For the branch, add the second tunnel and the BGP neighbor, verify the branch prefers the fibre path, then test failover during the window.
6. Validate: health-check status stable during an artificial load test, get router info routing-table all unchanged, session table shows NAT addresses consistent with the egress member, tunnel stays up under load, and a controlled member-down test moves only new sessions with acceptable impact.
7. Rollback: revert to the saved FortiManager revision and reinstall. Keep the old health-check profile as a named object so rollback is a one-line policy change.

Document that the real fix is capacity: the backup job saturating fibre is the root cause, so reschedule or rate-limit it, otherwise the same event recurs with different symptoms.