Monitoring, Backup, and Recovery
Monitoring, Backup, and Recovery
This domain tests detecting problems before users do and restoring service within agreed targets.
Core rules
- Four golden signals: latency, traffic, errors, saturation. Host metrics: CPU load versus utilization, memory including cache, disk IOPS and queue depth, network errors and drops.
- Collection: node_exporter/Prometheus, Zabbix, Nagios, Windows Performance Monitor, SNMP v3 (never v2c with the public community). Agents give richer data; SNMP is ubiquitous.
- Alert on symptoms that matter; every alert needs a runbook and an owner. Suppress flapping and tune thresholds to avoid fatigue.
- RTO is the maximum acceptable downtime; RPO is the maximum acceptable data loss. Backup design follows from these, not the reverse.
- 3-2-1: three copies, two media types, one off-site. Full plus incremental or differential chains; snapshots are not backups.
Worked example
RPO is 15 minutes and RTO is 2 hours for a database. Reasoning: nightly full backups give an RPO of 24 hours, which fails. Use continuous log shipping or 15-minute snapshots plus transaction log backups. For RTO, rehearse a restore: provision a host, restore the base backup, replay logs to the target time, run validation queries, then switch traffic. Measure elapsed time; if it exceeds 2 hours, automate steps until it does not.
Traps
- Backing up without testing restore - the only backup that counts is a restore.
- Keeping backups on the same SAN or array as production.
- Confusing a snapshot with an independent copy.
- Copying default alert thresholds without baselining normal behavior.