A failover exposed six weeks of configuration that never reached the standby

The chain: with Auto Sync disabled, synchronization is a deliberate action. It was happening because an engineer pushed after each change. When the firewall rule blocked the control-plane path between the devices, those pushes started failing. The local changes on A still succeeded, so the GUI and the change tickets looked normal. The device group moved out of sync and stayed there for six weeks. It surfaced only when B had to become active - and B's configuration was six weeks old.

Why auto sync would not have saved you either: auto sync is a convenience, not a guarantee. It only helps if the control path works, and a disabled auto sync in a change-controlled environment means the last successful push is the true state of the standby. Either way, the outage is caused by not verifying synchronization, not by the setting.

Evidence to collect:

• tmsh show cm sync-status and tmsh show cm device-group dg-ha on both devices - look for out-of-sync state, pending changes, and the last sync time.
• /var/log/ltm on both devices around the time of the firewall change: failed connection attempts from the peer.
• The firewall rule change record, to correlate date and time with the first failed sync.
• Configuration diff: list virtual servers on A and B (tmsh list ltm virtual one-line) and diff them; the fourteen missing objects should appear immediately.
• Confirm that no synchronization alarm or SNMP trap was configured - that is the real root cause of the six weeks of silence.

1. Restore the control path first. Reopen the required traffic between the devices, verify management reachability in both directions, and check that device trust and the device certificates are still valid (tmsh show cm cert). Synchronizing over a broken path just fails again.

2. Choose the authoritative device deliberately. A is authoritative here: it has been serving traffic and holds the fourteen objects that the business needs. Confirm that by diffing the object lists rather than assuming - if B had received emergency changes during the incident, you now have two divergent configs and one of them will be lost.

3. Fix time before syncing. B has drifted four minutes. Restart or force NTP on B and confirm it is tracking a source, because unsynchronized clocks cause confusing failures in logs and can affect synchronization and certificate validation.

4. Push the configuration. From A, run tmsh run cm config-sync to-group dg-ha (the GUI equivalent is Device Management › Overview › Full Sync › Push). A full sync overwrites B's configuration with A's - which is exactly what you want, and exactly why step 2 matters.

5. Verify, then save. tmsh show cm sync-status should report the pair in sync, and the object diff should be empty. Run tmsh save sys config on both devices so a reboot comes up with the synced config rather than a stale on-disk version.

6. Test. Force A to standby in an approved window, confirm the floating self IP and the fourteen virtual servers are present and answering on B, then fail back.

Detect it, do not hope. The single highest-value control is an alert on synchronization state: a periodic poll of show cm sync-status with an SNMP trap or an iCall script that fires when the device group leaves sync, plus a dashboard panel. Six weeks of silent drift becomes a same-day ticket.

Make the push part of the change, not an afterthought. Every change ticket that touches a synchronized object gets a verification step: push, confirm in sync, and record the result. If auto sync is disabled for change control, that discipline is the substitute for it and has to be enforced.

Reconsider the auto sync setting on its own merits. Enabled auto sync means a change on the active reaches the standby without a human step, at the cost of a bad change propagating instantly. A common compromise: leave auto sync enabled in the device group, keep changes on the active only, and rely on the sync-status alert to catch the failures. What does not work is disabled auto sync with no verification and no alert - which is the configuration that produced this incident.

Protect the control path. Put the inter-device synchronization traffic in the firewall's documented exception set, review it whenever the management-network rules change, and treat loss of that path as a P1 rather than a background warning.

Keep time honest. Monitor NTP offset on both devices; a four-minute drift means the alerting and log correlation you are relying on will mislead you.

Practise. Run a scheduled failover drill each quarter - brief, announced, and with the object diff checked before and after. A standby that has never carried traffic is a hypothesis, not a safety net.