Ensuring reliability and operational excellence
Ensuring reliability and operational excellence
This domain tests designing for reliability: SLOs, error budgets, redundancy, disaster recovery, and operational practices that sustain excellence.
Core rules
- SLOs and error budgets: define SLIs (availability, latency), SLO targets, and error budgets; alert on burn rate; balance feature velocity against reliability.
- Redundancy: deploy across zones (regional MIGs, regional GKE); multi-region for critical apps; global load balancing with health checks; graceful degradation; retries with exponential backoff and jitter.
- Data durability: Cloud Storage dual-region/multi-region; Spanner multi-region; Cloud SQL cross-region replicas; backups and point-in-time recovery; test restores.
- Disaster recovery: define RTO/RPO; warm standby or pilot light; automate failover with Cloud DNS routing policies and load balancer failover; run game days.
- Operational excellence: IaC, CI/CD, canary deployments, postmortems, chaos engineering (simulate zone failure); Cloud Monitoring SLO monitoring.
Worked example
A payment service requires 99.99% availability and RPO 5 minutes. Reason: deploy Spanner multi-region (or Cloud SQL with cross-region replica and automated failover); use a global external load balancer with health checks across two regions; set an SLO of 99.99% with error-budget alerts; configure Cloud DNS failover; run quarterly failover drills. This meets the requirement.
Traps
Confusing RTO and RPO; single-zone deployments for critical apps; not testing backups; setting SLOs without measurement; ignoring error budgets; over-engineering for 100% availability; forgetting that multi-region increases cost and complexity.