Failover Flapping and the Deployment That Would Not Land
A pair of FTD 2140s in active/standby, managed by FMC 7.2, protects a campus data center. Overnight the pair failed over three times: unit A to unit B at 02:14, back to A at 02:41, and to B again at 03:02. Users reported brief outages each time and several long-lived database sessions were reset. FMC Health Monitor shows Ethernet1/2 going down and back up on whichever unit was active roughly twenty seconds before each failover. The upstream switch logs show the corresponding port transitioning down with no administrator action. At 05:30 a scheduled policy deployment of a new access control policy reports success for unit B and failure for unit A with a communication timeout, even though 'show managers' on unit A reports Connected. The pair is now serving traffic from unit B and the customer wants the deployment finished before the business day starts.
1. What is the most likely mechanism behind the repeated failovers, and what do you check first?
Read the worked answer
The failover trigger is interface failure, not a device crash. When an interface that is monitored in the HA configuration goes down on the active unit, FMC-driven HA treats that as a failure condition and hands the active role to the peer. Because Ethernet1/2 flapped on whichever unit was active, the common element is the physical link and the switch, not either chassis.
Check, in this order:
- 'show failover' on both units for the reported last failover reason; it should point at the interface rather than a peer timeout.
- Interface counters on Ethernet1/2 for CRC, input and output errors, and runts. Rising CRC errors point at cabling, SFP or duplex/speed negotiation.
- Physical layer: SFP diagnostics, cable reseat or swap, and whether the port negotiated the expected speed and duplex.
- Upstream switch logs and stats for the same port, including whether the link drops when the other unit's port comes up, which suggests a spanning-tree or port-configuration interaction rather than a bad optic.
The temptation is to disable interface monitoring and be done. Resist it: that only removes the alarm and leaves you blind to a genuinely dead inside link, which is a far worse failure than a failover.
2. If the physical fault cannot be fixed before business hours, what interim mitigation is defensible and how do you frame the risk?
Read the worked answer
The defensible interim step is to stop monitoring Ethernet1/2 in the HA configuration while leaving the other interfaces monitored, so a single flapping port no longer moves the active role. This keeps detection of a failed outside link and keeps the pair stable for the business day.
Frame the risk explicitly in writing:
- With Ethernet1/2 unmonitored, a real failure of that inside link will not cause a failover. Traffic may be black-holed on the active unit while the pair still reports healthy, and only connection failures will reveal it.
- The mitigation is temporary and must carry an owner and an expiry, tied to the cable or optic replacement.
- Monitoring different interfaces than the hardware team expects creates confusion later; note the deviation in the device documentation, not just the ticket.
Also verify that the traffic resets during failover were genuinely caused by the role change and not by a failover link that is too small or shared: stateful failover only preserves sessions if the failover link is healthy and dedicated, and a congested failover link is itself enough to break long-lived database sessions. Do not change preemption or preferred-role settings to paper over a physical problem; they change who is active, not whether the link is stable.
3. Why did the deployment succeed on unit B and time out on unit A, and what is your sequence to finish it safely?
Read the worked answer
In an HA pair FMC deploys the configuration separately to each unit over that unit's own management address on TCP 8305. A successful deploy to B and a timeout to A means the policy is fine but A's management path or A's readiness is not: a unit that has just been through repeated role changes can be in an unsettled HA state, and 'Connected' in 'show managers' only proves the sftunnel session is up, not that the unit is ready to accept and apply a configuration.
Sequence:
- Read the FMC deployment task detail and confirm which unit was reported as unreachable; the summary line alone hides this.
- On unit A, run 'show failover' and 'show interface' to confirm the HA state and that its management interface and default route are intact. Look for an HA state that is not the expected standby role after flapping.
- Check reachability from A to the FMC on 8305, and from the FMC to A's management IP, ideally by capture rather than assumption.
- Do not deploy again while A is unstable. Stabilise the HA state first (resolve the interface issue, resume the unit if it is suspended), confirm both units agree on active/standby, then redeploy and watch both units.
- After success, verify both units report the same policy version and that the standby has the new configuration staged, otherwise the next failover silently reverts the policy.
Finish the deployment only after the HA pair is genuinely stable; a half-deployed pair is more dangerous than an unapplied policy.