HA Path Monitoring Flapping at a Hospital
A regional hospital deploys a PA-3410 HA pair in active/passive mode at its primary site. The pair sits behind two ISP routers running HSRP; each firewall has one ISP-facing interface in a separate virtual router. HA1 runs over a dedicated 1 Gbps link, HA2 over a 10 Gbps link, and HA3 over a dedicated 1 Gbps link. Path monitoring uses five targets: the HSRP virtual IP, two public DNS resolvers, the ISP router loopback, and the MPLS provider edge loopback. Failure condition is set to any, failure threshold is 3, and interval is 5 seconds. Since cutover, the pair has failed over four to six times per day, usually for 30 to 90 seconds. The ISP reports no circuit outage. Users on the MPLS side see brief drops during each failover; internet-side users do not. The hospital requires 99.99% uptime and cannot accept unnecessary failovers.
1. Diagnose the most likely cause of the repeated failovers and identify the specific HA settings you would change.
Read the worked answer
The pattern points to false-positive path monitoring, not a real ISP outage. With failure condition set to any, a single monitored target that is intermittently unreachable triggers failover. Public DNS resolvers are poor targets because they may rate-limit ICMP or be reachable only through a route that changes during normal BGP or MPLS events. The HSRP virtual IP may also not answer ICMP from the firewall depending on HSRP configuration and source address selection.
I would reduce the path group to stable, directly relevant next hops: the ISP router loopback and the MPLS PE loopback, both tested with BFD where supported. Change failure condition to all so both paths must fail, raise the threshold to 10, and use a 3-second interval with a longer down timer. Enable link monitoring on the physical interfaces as a separate, faster signal. Disable preemption unless there is a documented need, and verify that the MPLS virtual router is fully synchronized on HA3. Finally, confirm that the monitor source IP is permitted by the upstream devices and that ICMP is not being policed.
2. Explain why MPLS-side users experience drops while internet-side users do not.
Read the worked answer
The asymmetry suggests the failing monitor is in the MPLS virtual router or that the MPLS path is not fully present on the standby. When the active firewall fails over, internet-side users may recover through a static default route and NAT session synchronization, while MPLS-side traffic depends on a separate routing table, BGP or static PE routes, and interface state that may not be synchronized or may take longer to converge. If the MPLS PE loopback is one of the monitored targets and it flaps, the firewall fails over even though the internet path is healthy. During that interval, MPLS sessions can be dropped because the standby lacks the current route or because return traffic arrives at the newly active node before the MPLS neighbor is re-established.
I would verify HA3 synchronization for the MPLS virtual router, check whether the MPLS interface and routes exist on the standby, and compare session tables before and after failover. A packet capture on both firewall nodes during a controlled failover will show whether the drop is routing convergence, session loss, or path-monitor-induced failover.
3. What test plan would you use before enabling preemption or returning the pair to production?
Read the worked answer
I would run a controlled test in a maintenance window with active SIP and HTTPS sessions crossing both the internet and MPLS paths. Steps: record HA state, path-monitor status, routing tables, and session counts on both nodes; force a manual failover; measure outage duration for each path; confirm that the floating IP moves and that BGP or static routing reconverges; and capture packets on client and server sides to distinguish session loss from routing loss. Then repeat with one ISP link shut down and with the MPLS PE loopback filtered, confirming that the revised monitor does not fail over unless both required targets are down.
Preemption should remain disabled until the team can show that the preferred node always has complete routing and session state before it takes over. I would also add alerting on HA state changes and path-monitor transitions so that future flaps are visible before users report them.