FTD HA Pair Flaps After a Switch Stack Reboot and Drops Trading Sessions
Two FTD 3105 appliances run as an active/standby pair managed by an on-premises FMC that also manages 22 other FTDs. During a switch stack reboot in the data center, the pair failed over twice within ten minutes. Approximately 3,000 established TCP sessions to a trading application were reset, even though stateful failover is configured. The failover and stateful failover interfaces are VLANs carried on the same switch stack that rebooted. Interface monitoring is enabled on the inside and outside interfaces. After the event, the FMC shows one unit as active and the other as standby ready, but the two units disagree on the interface monitoring configuration, and the standby unit has not received the last two policy deployments. The business wants to know why sessions dropped and whether this can happen again during the next maintenance window, which is in nine days.
1. Explain why the failover occurred twice rather than once, and why the configuration disagreement on interface monitoring matters here.
Read the worked answer
The first failover is defensible: the switch stack rebooted, so the active unit lost its monitored interfaces or its failover communication, and the standby took over. The second is the real problem.
When the stack came back, both units re-established their monitored interfaces at slightly different times, and the failover hello and unit hold timers are short by design - they must be, to keep takeover fast. If the restored unit's interfaces come up before the switch has fully converged (STP forwarding, link aggregation, ARP), the unit can declare itself healthy, see the peer as failed, and take over again. That is a flap, and a flap is worse than a single failover because each transition resets the connection table and the state replication stream.
The interface monitoring disagreement is directly relevant: if the two units evaluate different interfaces as monitored, they make different decisions about who is healthy. One unit can believe it is the only viable unit while the peer believes both are healthy. You cannot reason about failover stability until both units have identical monitoring configuration, because the failover algorithm is a comparison of two health states. Any drift here also suggests the standby has not been receiving configuration - which the missed deployments confirm - so this is a sync problem, not just a flapping problem. Fix the sync first: whatever the standby is running is not what the active is running, and a failover right now would move production traffic onto a configuration nobody has reviewed.
2. Why were 3,000 established sessions reset despite stateful failover being configured, and what would you check on the devices to confirm?
Read the worked answer
Stateful failover preserves sessions only under specific conditions, and this scenario breaks several of them.
First, state replication needs a working stateful path between the units. Both the failover link and the stateful failover link were VLANs on the stack that rebooted, so during the event there was no replication path at all - the standby could not receive state for connections established or changed while the link was down. Sessions created in that window exist only on the active unit.
Second, a double failover means two role transitions in ten minutes. Each transition resets the connection table on the newly active unit for anything not replicated, and the trading application's long-lived sessions are exactly the ones most likely to be caught mid-transition.
Third, if the monitored interface state is inconsistent between the units, some sessions may have been handled asymmetrically during the transitions, so the return traffic did not follow the same unit.
Confirm on the devices: check the failover state and the stateful failover status on both units, check whether the stateful link was up during the event and whether any replication errors or queue drops are recorded, compare connection counts before and after, and review the logs for the exact timestamps of the two transitions against the stack's port-up times. If the stateful link was down or saturated during the reboot, that is the finding - the configuration was correct but the transport was not available when it mattered.
3. What changes do you make before the next maintenance window, and how do you verify the pair is genuinely ready?
Read the worked answer
Three structural changes and one procedural one.
Structural: move the failover and stateful failover links off the switch stack that carries production data, or at minimum onto ports served by a different stack member and a different power domain. The event was not caused by a failover misconfiguration - it was caused by the failover transport sharing fate with the data path, which defeats the purpose.
Structural: reconcile the configuration between the units. Resolve the interface monitoring disagreement in the FMC so both units have identical monitored interfaces, and re-evaluate whether monitoring both inside and outside at tight hold times is appropriate for this environment; aggressive monitoring plus slow switch convergence is what produced the second transition. Consider extending the interface hold time so the pair waits for the switch to converge instead of reacting to a transient.
Structural: resolve the missed deployments so the standby is running the same policy as the active, and investigate why two deployments failed - a repeating failure will reappear at the worst time.
Procedural: schedule the failover test inside the maintenance window, in a controlled way, with the business aware. Verify by forcing a manual failover and confirming: the transition happens once, both units report consistent state, existing sessions survive, the new active has the current policy, and logs and syslog export continue from the new active. Also record the timing of the transition so the next window has a baseline, and confirm FMC and both FTDs are synchronized to the same reliable time source, because every timestamp-based conclusion above depends on it.