Reliability, Resilience, and Disaster Recovery
Reliability, Resilience, and Disaster Recovery
This domain covers service level objectives and error budgets, failure-domain analysis, the DR pattern ladder, and - critically - how you prove any of it works. Scenarios typically state an RTO and RPO with a budget constraint and expect you to pick the cheapest pattern that satisfies both.
SLOs and error budgets
- An SLI is the measured indicator (success rate, p99 latency); an SLO is the target over a window (99.9% over 30 days); the error budget is 1 minus the SLO.
- The error budget policy is a governance agreement: when the budget is exhausted, feature releases pause and reliability work takes priority.
- Alert on user-visible symptoms using burn rate, not on raw resource thresholds that fire without customer impact.
Failure domains and patterns
- Zone, then region, then account or project, then global control plane. Each level needs its own plan and its own credentials.
- The DR ladder: backup and restore (hours), pilot light (minutes to hours), warm standby (minutes), active-passive multi-region, active-active (seconds, highest cost and complexity).
- Pick the cheapest rung that meets both RTO and RPO; anything more is spend without benefit.
- Resilience primitives: health checks with automatic replacement, retries with exponential backoff and jitter, circuit breakers, bulkheads, queue-based load leveling, graceful degradation.
Worked example
RTO 60 minutes, RPO 5 minutes, budget allows roughly 1.6x single-region cost. Backup and restore gives an RPO in hours and fails. Active-active gives an RPO near zero but costs 2.5–3x and exceeds the budget. Warm standby fits: the full stack runs at minimum capacity in a second region, asynchronous replication keeps lag under five minutes, and a global load balancer performs scripted failover. Pre-test the post-failover scale-up so the RTO lands at 20–40 minutes. Rehearse quarterly - an unrehearsed failover usually blows the RTO on the first real attempt.
Traps that cost marks
- Stating an RPO without a way to measure replication lag against it.
- A DR plan that depends on the same identity or control plane that failed.
- Retries without backoff, turning a transient blip into a retry storm.
- Never failing over on purpose; game days and controlled experiments are the only credible evidence.