HA Failover with an Incomplete ADOM Database
Two FortiManager 7.4.3 appliances form an HA pair in different data centers. The primary manages 40 FortiGates in ADOM 7.4 and receives policy changes daily. During a storage failure, the primary reboots and the secondary becomes primary. The old primary is powered off for repair. The new primary GUI shows all ADOMs and devices, but three policy packages are missing revisions from the last two days. Two installations to production FortiGates fail with policy package not found, and one installation succeeds but pushes an older policy. get system ha status reports the node as Primary, and diagnose fmgsync status reports that synchronization is not complete. The team has a full FortiManager backup from 36 hours ago and a partial export of one ADOM from 12 hours ago. They need to restore service without pushing inconsistent policy.
1. Why does the new primary have stale policy data, and what should be checked before any install?
Read the worked answer
FortiManager HA is primary/secondary with asynchronous database replication. The secondary can be promoted while its ADOM database is still behind; it will still answer GUI queries and show devices, so the GUI is not proof of consistency. The install failures and the older policy are symptoms of stale policy packages and revisions. Check get system ha status for role and peer state, diagnose fmgsync status for sync completion, and compare ADOM revision numbers against the last known change log. The message that synchronization is not complete is the key: do not install from this node. The fact that one install succeeded is worse than a failure because it pushed an older policy to production.
2. Given the available backups, what is the safest recovery path and what are its trade-offs?
Read the worked answer
Stop policy installs immediately. If the old primary can be repaired and brought back, do that first; let it become primary and allow HA synchronization to repopulate the secondary, because it holds the missing revisions. If it cannot be recovered, restore the 36-hour full backup to the new primary, then reapply the changes from the 36-hour window using the change log and the 12-hour ADOM export for the one ADOM it covers. The partial export cannot restore the other ADOMs, so it is a reconciliation aid, not a full recovery. After restore, verify ADOM revisions, run Install Preview rather than install, and only then install to a canary device. Accept that some recent changes may need manual re-entry; that is safer than pushing a stale package to 40 devices.
3. What monitoring and process controls would catch this before a production install?
Read the worked answer
Alert on diagnose fmgsync status and on HA role changes; do not treat Primary as synchronized. Schedule full FMG backups and export ADOM revisions after major changes. Before maintenance, confirm synchronization is complete and avoid deliberate failover during a policy change. Use a canary device and Install Preview for high-risk changes. Record the ADOM revision and package version in the change ticket so a post-failover node can be validated. If the platform supports it, monitor HA synchronization via SNMP and test failover in a lab. The goal is to detect an incomplete synchronization before an install, not after a production device has received an old policy.