DRBD Split-Brain After a Switch Maintenance Window
Two nodes, db1 and db2, run Pacemaker and DRBD 9.1. A resource r0 backs a 2 TB ext4 filesystem and is configured active/passive with protocol C. Replication uses a dedicated 10 GbE link; cluster heartbeat uses a separate 1 GbE bond. During a switch firmware upgrade, the 10 GbE link flapped repeatedly. Pacemaker did not fence because the heartbeat bond stayed up, stonith-enabled was true, but the fence agent credentials had expired. An operator ran drbdadm primary r0 on db2 to mount the filesystem for an urgent report. Writes were accepted on both nodes for about two hours. drbdsetup status r0 now shows StandAlone on both, Primary/Secondary on db1, and Primary/Secondary on db2. Application logs show order IDs up to 88421 on db1 and 88437 on db2, but there is no shared sequence.
1. What must happen before any DRBD connect or discard command is issued, and how do you decide whether db1 or db2 is authoritative?
Read the worked answer
Treat this as data reconciliation before replication repair. Stop application writes and unmount the filesystem on both nodes. Do not run drbdadm connect yet, and never use --discard-my-data on both sides. Capture drbdsetup status r0 --verbose, filesystem metadata, and application logs. Because db2 has order IDs through 88437 while db1 has 88421, db2 is a candidate, but only if the application confirms that no db1-only records were created. If both sides contain unique records, DRBD cannot merge them; export both datasets and repair at the application layer, or restore from backup. If db2 is authoritative, record the decision and the exact recovery point before changing DRBD state. The critical judgement is that DRBD resynchronization fixes block-level divergence, not business-level lost writes.
2. Assuming db2 is authoritative, give the DRBD recovery sequence for db1 and db2, including verification steps.
Read the worked answer
On the loser, db1, run drbdadm disconnect r0, then drbdadm secondary r0 if it is still primary. On db2, keep r0 primary and run drbdadm disconnect r0 to clear the stand-alone state. Reconnect the winner first with drbdadm connect r0; on db1 use drbdadm connect --discard-my-data r0. Watch drbdsetup status r0 until cs:Connected and ro:Primary/Secondary, then run drbdadm verify r0. Do not mount the filesystem or start Pacemaker resources until the verify completes and the application has a consistency check. If db2 is not authoritative, invert the roles. After recovery, test fencing and correct the expired credentials so a future replication-link failure cannot leave both nodes primary.
3. What configuration and operational controls would you add to prevent a repeat, given that the heartbeat stayed up while replication failed?
Read the worked answer
Separate replication health from cluster membership in monitoring. Pacemaker may see healthy nodes while DRBD is in StandAlone, so alert on drbdadm cstate and drbdsetup status in addition to pcs status. Keep allow-two-primaries no, fix and test the fencing credentials, and ensure the fence device can actually remove the failed node from the replication path. Consider a Pacemaker resource or external script that stops the filesystem and demotes DRBD when the replication link has been down beyond a defined threshold. Schedule drbdadm verify runs and review their output. Avoid manual drbdadm primary during an incident; require the cluster to promote and require confirmation that the peer is not primary. Finally, test a replication-link failure in a maintenance window and record whether fencing, demotion, or both occur.