Quorum Loss and the No-Quorum Policy in a Split Data Center
Five nodes run Pacemaker 2.1 and Corosync 3 on RHEL 9: ha-a1 and ha-a2 in rack A, ha-b1 through ha-b3 in rack B. A PostgreSQL resource, a VIP, and a filesystem are configured as a group. Fencing was set to false two years ago because the team could not obtain IPMI access, and no-quorum-policy was changed to ignore during an earlier two-node pilot. The inter-rack link is taken down for 30 minutes to replace an optic. During the window, crm_mon on each side reports a partition; ha-a1 runs the PostgreSQL primary and VIP, and ha-b1 also runs a PostgreSQL primary and has claimed the same VIP on its local segment. Both databases accepted writes for 18 minutes before the link was restored. No shared LUN is involved; each node has local PostgreSQL data and asynchronous streaming replication was configured but not enforced.
1. Explain why the minority rack continued to run PostgreSQL and the VIP. Which effective configuration values and cluster states would you inspect before calling this a Corosync failure?
Read the worked answer
Inspect the effective cluster properties first: pcs property show no-quorum-policy, pcs stonith status, and crm_mon -Af. With no-quorum-policy=ignore, Pacemaker deliberately keeps resources active when the partition loses quorum; it does not treat quorum loss as a stop condition. Because stonith-enabled=false, the minority also has no self-fencing path that would isolate ha-a1 before it writes. The resource group meta attribute requires should also be checked; if it is not set to quorum, the group is not tied to quorum state. The partition report alone is therefore not a bug: the configuration allowed both sides to believe they were entitled to run PostgreSQL and the VIP. The duplicate VIP and two writable primaries are the predictable result, not evidence that Corosync failed. A proper diagnosis separates transport failure from policy failure.
2. The link is back and both nodes now have divergent PostgreSQL data. What is the immediate recovery sequence, and how do you choose which side to trust?
Read the worked answer
First isolate ha-a1 from clients and the network so it cannot accept more writes; do not simply restart Pacemaker, because that can trigger another promotion. Capture pg_controldata, current WAL LSN, and application sequence logs from both nodes. Choose the authoritative side only after application-level reconciliation: ha-b1 may have later order IDs but may lack IDs unique to ha-a1, so neither LSN alone proves completeness. If the application can replay or merge, rebuild the rejected node with pg_basebackup and re-establish streaming replication; otherwise restore from the last verified backup. Prevent recurrence by enabling real fencing with IPMI or a PDU and testing it, setting no-quorum-policy=stop, adding a qdevice at a third site, and giving the resource meta requires=quorum where appropriate. Keep the inter-rack link redundant, and alert on crm_mon partition and dual-primary states.