High Availability Cluster Storage
High Availability Cluster Storage
This domain tests shared block storage and cluster filesystems: iSCSI, device-mapper multipath, cluster LVM, DLM, GFS2/OCFS2, and DRBD. The central rule is that only one node may update shared metadata, and no node may mount shared data until locking is active.
Core rules
- Reach shared LUNs through redundant paths. Enable
multipathd, define policies in/etc/multipath.conf, identify WWIDs withmultipath -ll, and use/dev/mapper/<wwid>rather than/dev/sdX. - Local LVM is unsafe on shared disks. Use cLVM (
clvmd, older) orlvmlockdwith DLM so metadata changes are serialized. In Pacemaker, order DLM before cLVM before the filesystem. - GFS2 and OCFS2 are cluster-aware. GFS2 needs DLM and a lock table (
cluster:fs); create one journal per node withmkfs.gfs2 -p lock_dlm -t webcluster:gfs -j 2. Mount through aFilesystemresource, not fstab. - DRBD is network RAID1. Pacemaker promotes one node. After split-brain, disconnect both, then use
drbdadm -- --discard-my-data connecton exactly one side. - Fence first: a node must be fenced before its locks are released.
Worked example
Two nodes share a multipath LUN. Use multipath -ll to get /dev/mapper/mpatha; pvcreate /dev/mapper/mpatha; vgcreate vg_shared; lvcreate -L 100G -n gfslv vg_shared. Create DLM (ocf:pacemaker:controld) and cLVM (ocf:heartbeat:clvm), ordering DLM then cLVM. Format mkfs.gfs2 -p lock_dlm -t webcluster:gfs -j 2 /dev/vg_shared/gfslv; add a Filesystem resource with fstype=gfs2, device=/dev/vg_shared/gfslv, directory=/mnt/gfs. Reasoning: DLM supplies locks, cLVM serializes metadata changes, GFS2 uses the lock table, and two journals cover both nodes.
Traps
Creating a local VG on a shared disk corrupts metadata. Mounting GFS2 manually on both nodes bypasses fencing. Omitting -p lock_dlm creates a local filesystem. Reconnecting DRBD after split-brain without discarding on one side re-creates the split brain. Forgetting to enable multipathd leaves a single path.