QDevice Firewall Change Removes the Last-Man-Standing Tiebreaker
A two-node Pacemaker 2.1 cluster, web1 and web2, uses a Corosync qdevice hosted on qnetd.example.net to provide a tiebreaker. The cluster runs a VIP and HAProxy in an active/backup arrangement. A security team tightens the host firewall on qnetd and blocks TCP 5403. No one notices because both cluster nodes remain up and pcs status is clean. Two days later, web1 panics and reboots. web2 remains healthy but stops the VIP and HAProxy; corosync-quorumtool -s on web2 shows Nodes: 1, Expected votes: 3, Highest expected: 3, Total votes: 1, Quorum: 2 Activity blocked. The team asks whether to run pcs quorum expected-votes 1 or remove the qdevice so the VIP can fail over.
1. Why did the qdevice failure remain invisible until web1 panicked, and why did web2 stop resources instead of taking over the VIP?
Read the worked answer
With both nodes up, web1 and web2 each have one vote; the qdevice supplies the third vote, so quorum is 2 of 3. The loss of qnetd does not stop the cluster while both nodes are alive because the two node votes still meet quorum. When web1 panics, web2 has one vote and cannot reach the qdevice, so total votes are 1 and quorum is 2. Pacemaker stops resources because running the VIP alone could create two active HAProxy instances after a network partition. This is correct quorum enforcement, not a Pacemaker bug. Lowering expected votes to 1 or removing the qdevice would make each node believe it can run alone, which is exactly the split-brain risk the qdevice was added to prevent.
2. What is the safe remediation, and under what conditions, if any, would you change quorum expectations?
Read the worked answer
Restore the qdevice path first. On qnetd, allow TCP 5403 from both web1 and web2; on the nodes, verify corosync-qdevice is running and pcs quorum device status shows the device. Confirm with corosync-quorumtool -s that expected votes are 3 and quorum is 2, and that web2 regains quorum before starting resources. If qnetd is permanently gone, do not simply set expected-votes 1 on a two-node cluster; add a third Corosync node or a replacement qdevice, or accept that a single node cannot safely run and keep the service down. Fencing can help only if it reliably isolates the failed node; it does not replace quorum. Changing expected votes is acceptable only as a short, controlled action when you can prove the other node is dead and cannot return, not as a convenience to restore service.