Recovering from a Failed HA Node and a Bad Restore

Two separate mistakes compounded.

First, restoring a 01:00 backup onto the repaired node discarded seven hours of changes that the standby had accepted after failover. The standby was the authoritative node; the repaired node should have been rebuilt to the same version and joined as a standby so it could receive a full sync from the active, not restored from backup.

Second, the node was re-added while its database was stale and its HA state was not clean, producing split-brain symptoms: both nodes claim active, replication is broken, and data differs.

Correct sequence now:

1. Stop and isolate. Prevent both nodes from accepting management writes; do not let automation or operators make changes during recovery. Identify which node has the most recent authoritative data (the standby, including the 07:30 changes).
2. Do not restore the 01:00 backup to the standby. That would propagate the data loss to the only good copy.
3. Rebuild the repaired node to the same XIQ-SE version and patch level, with matching certificates, time, and license, and rejoin it as a standby so it syncs from the authoritative node.
4. Validate replication status, HA state, device counts, and a sample of 07:30 changes before returning to service.

If the authoritative node is itself corrupted, then a coordinated restore of both nodes from the same backup is the fallback, with the data loss documented.

Build a pre-join checklist and treat it as a gate:

• Version and patch parity. A different XIQ-SE build or patch level will fail replication or produce undefined behavior.
• Identity and certificates. Hostnames, management IPs, server certificates, and trust relationships must match the design. A restored node may carry an old certificate that the peer rejects.
• Licensing. Confirm the node is licensed for its HA role; an unlicensed or partially licensed secondary may not take over correctly.
• Time synchronization. Check NTP on both nodes. Clock skew breaks authentication and log correlation.
• Database state. The joining node should be clean or explicitly synchronized from the active, never carrying a stale independent copy.
• HA heartbeat and network path. Verify the dedicated HA/heartbeat link and the management path, including any firewall rules between nodes.
• Backup and change freeze. Take a fresh backup of the authoritative node before the join, and freeze unrelated changes until replication is healthy.

Finally, record the actual state (active node, last change time, device count) in the ticket. The failure here was not only technical; it was a recovery performed without a known-good reference.

The incident exposes a process gap, not just a technical one.

• Define the backup scope explicitly. Capture the database, application configuration, licenses, certificates, and any custom reports or scripts. A backup that omits certificates will produce a node that restores but cannot establish trust.
• Test restores quarterly into an isolated environment, and record the actual restore time. An untested backup is a hypothesis.
• Write the HA recovery runbook. It should say: identify the authoritative node, never restore a stale backup over a newer active node, rebuild and rejoin, validate, then unfreeze. Include the decision point for a coordinated restore if both nodes are bad.
• Protect the backup repository. Restrict who can restore, require a second approval for production restores, and log restores in the change system.
• Set RPO/RTO expectations. The 01:00 backup implies up to eight hours of data loss if used as the recovery point; the HA design should make that unacceptable except in a disaster.
• Post-incident review. Capture the 02:10 failover timeline, the 09:00 actions, and the missing 07:30 changes, and feed the findings into the runbook before the next quarter.