HA Preemption and Stale Configuration After a Power Event
Two FortiMail 7.2 appliances at a data center run HA active-passive. FM-A at 10.60.0.10 is the preferred primary, FM-B at 10.60.0.11 is secondary, and the mail VIP is 10.60.0.12; MX points to the VIP. During scheduled power maintenance, both units were shut down. FM-B booted first and became primary. When FM-A booted 20 minutes later, it also claimed primary, and both units accepted mail for the same domain for several minutes. Users received duplicate messages, and some messages were delivered twice to Exchange. An administrator had changed an inbound policy on FM-A while FM-B was offline; after the event, HA status reports configuration out of sync. The nightly configuration backup is sent to an FTP server, but the last three backups failed after a firewall rule change. The team needs to restore a single active node, preserve the changed policy, and improve failover and backup procedures.
1. What caused both nodes to become active, and what would you check and change in the HA configuration?
Read the worked answer
This is a split-brain symptom: both nodes believed they were primary because they could not confirm each other's state when FM-A booted. Check the HA heartbeat interfaces and their switch VLANs, the HA status on both units, priority and preemption settings, and whether link monitoring declared the peer dead. If the heartbeat path was down or delayed, FM-A may have started as primary without seeing FM-B. Configure a dedicated heartbeat path that is independent of the mail data path, verify that both nodes use the same HA group and encryption settings, and align priorities so the intended primary wins when communication is healthy. Consider boot ordering or a startup delay so the secondary does not grab primary and then fight preemption. To recover, isolate one node, confirm its configuration and queue, then bring the peer into HA and verify a single primary before restoring the VIP.
2. How do you recover without losing the inbound policy change and without creating more duplicates?
Read the worked answer
First identify which node has the wanted configuration: FM-A contains the inbound policy change, so it must be the source of truth. Back up FM-A's configuration immediately before making further changes. Put FM-B into a maintenance or non-HA state and prevent it from accepting mail on the VIP. Force a configuration synchronization from FM-A to FM-B, or restore FM-B from the FM-A backup if synchronization will not complete, then verify the policy is present and HA status is consistent. Inspect both mail queues and delivery logs before reenabling the VIP; duplicate messages already delivered cannot be recalled, so document them and check message tracking for additional duplicates. Do not synchronize from stale FM-B to FM-A, which would overwrite the policy change. Only after both nodes agree should you restore normal HA and monitor delivery for recurrence.
3. What backup and monitoring changes would you make after this event?
Read the worked answer
Fix the backup path first: move from unencrypted FTP to SFTP or SCP with a dedicated account, update the firewall rule, and test the transfer. Schedule configuration backups after approved changes as well as nightly, retain multiple versions, and test a restore on a non-production unit. Understand that configuration backups do not preserve messages already queued or quarantined, so plan separately for mail queue and quarantine recovery and avoid firmware upgrades without a current configuration export. Monitor backup success, HA state, heartbeat status, queue size, and failover events with alert email, syslog, SNMP, or FortiAnalyzer, and route alerts to an on-call address. Add a documented failover and recovery runbook that includes isolating a split-brain node, identifying the authoritative configuration, and verifying a single primary before the VIP is reactivated.