LPIC-3 High Availability and Storage Clusters cheat sheet

Corosync provides cluster membership and messaging; Pacemaker's votequorum decides partition eligibility. Quorum requires more than half of configured votes, except two_node mode.

Pacemaker resource agent classes include OCF, LSB, systemd, service, upstart, and nagios; OCF agents expose parameters and support clone and promotable resources.

STONITH fencing isolates a failed node before resource recovery; stonith-enabled=true is required when resources use shared data, and a watchdog can self-fence a node.

Constraint scores: positive prefers, negative avoids, INFINITY enforces, -INFINITY forbids; a location constraint with -INFINITY bans a resource from a node.

An order constraint's mandatory flag requires the first resource to start before the second and stop after it; symmetric=true also reverses the stop order.

A colocation constraint with score INFINITY requires resources to run on the same node; a negative score prefers separation, and -INFINITY forbids co-location.

pcs cluster standby moves managed resources off a node while cluster services remain active; pcs resource maintenance freezes resource actions without stopping services.

A corosync-qdevice adds one vote through qnetd, helping a two-node cluster keep quorum after a node loss; it does not provide fencing.

Failed monitor actions follow the resource's on-fail policy: restart, fence, standby, or block; pcs resource cleanup clears a failed state after repair.

DM-Multipath aggregates multiple paths to a LUN into one device; multipath -ll displays path state, and path_grouping_policy, prio, failback, and no_path_retry control selection and queueing.

iSCSI target exports LUNs by IQN; the initiator discovers targets, logs in, and creates SCSI devices; CHAP authenticates discovery and session, and multipath requires separate fabrics or ports.

LUN masking limits a LUN to selected host WWNs or IQNs on the array; fabric zoning limits port-to-port visibility; both layers are needed for correct access control.

GFS2 requires a cluster lock manager (DLM), quorum, and fencing; mkfs.gfs2 -p lock_dlm -t cluster:fs creates it, and all nodes mount the same shared block device.

OCFS2 uses a cluster stack, slot map, and fencing; tunefs.ocfs2 manages parameters, and concurrent mounts require the same cluster name and heartbeat configuration.

lvmlockd coordinates shared VG metadata with sanlock or DLM; it replaces clvmd, requires use_lvmlockd=1, and shared VGs are created with vgcreate --shared.

SCSI-3 persistent reservations register and reserve LUN access for cluster nodes; they prevent a partitioned node from writing after fencing and are used by cluster filesystems and agents.

A shared VG must not be activated exclusively on multiple nodes; only one node owns normal activation, while cluster filesystems use lock managers for concurrent access.

Fencing occurs before GFS2/OCFS2 recovery; without successful fencing, the surviving nodes cannot safely replay journals or release locks.

Ceph monitors maintain the cluster maps and form a quorum; use an odd number, typically three or five, and a monitor loses quorum if it cannot reach a majority.

Ceph OSDs store objects and handle replication, recovery, and backfill; each OSD belongs to a CRUSH failure domain, and OSD down/in state affects data placement.

CRUSH maps objects to placement groups and PGs to OSDs using buckets, weights, and rules; the failure domain, such as host or rack, controls replica separation.

A placement group is a logical pool partition; target PG count is approximately (OSD count × 100) / replica count, rounded to a power of two, with pg_num and pgp_num equal.

Ceph RBD images are RADOS objects exposed as block devices via librbd; snapshots and clones are copy-on-write, and exclusive-lock supports journaling and live migration.

CephFS requires at least one MDS; it uses a metadata pool and data pool, and clients mount it as a POSIX filesystem via the kernel client or FUSE.

GlusterFS volumes are built from bricks, which are exported directories; volume types include distributed, replicated, dispersed, and combinations, with quorum preventing split-brain writes.

DRBD protocol C acknowledges a write only after local and remote disk writes complete; protocol A is asynchronous, and protocol B acknowledges after remote memory receipt.

DRBD split-brain is resolved by selecting a source, discarding or merging the divergent side, and reconnecting; dual-primary mode requires quorum and fencing to avoid concurrent writes.

RAID 1 mirrors data and survives one disk failure per mirror; RAID 5 uses one parity disk and survives one failure; RAID 6 uses two parity blocks and survives two failures.

RAID 10 stripes mirrors; it tolerates one disk per mirror but not two disks in the same mirror; usable capacity is half of raw capacity for two-disk mirrors.

mdadm --create builds arrays, /proc/mdstat shows status, and --fail, --remove, --add replace a failed disk; --detail prints UUID, level, state, and component devices.

Linux bonding mode 1 is active-backup, mode 4 is 802.3ad LACP, mode 5 is balance-tlb, and mode 6 is balance-alb; miimon sets link monitoring interval in milliseconds.

smartctl -a prints SMART attributes, -H prints overall health, and -t long starts an extended self-test; reallocated and pending sector counts indicate media degradation.

kdump reserves crashkernel memory, boots a capture kernel after panic, and writes vmcore to /var/crash; the crash utility analyzes vmcore with the debug kernel.

A watchdog timer resets a node if software stops refreshing it; /dev/watchdog is refreshed by a daemon, and systemd RuntimeWatchdogSec sets the hardware watchdog interval.

NUT's upsd serves UPS status and upsmon triggers shutdown on low battery; a UPS plus dual power supplies protects a single node from utility power loss.

ECC memory corrects single-bit errors and detects many multi-bit errors; IPMI SEL logs hardware events, and patrol scrubbing detects latent memory errors.