SRE Foundation cheat sheet

SRE applies software engineering to what was traditionally called operations: reliability is delivered through code, automation, and measurement rather than manual effort.

Fifty percent rule: an SRE spends at most 50% of time on operational work (toil, tickets, on-call) and at least 50% on engineering that reduces future work.

SLOs are the primary control mechanism: reliability targets, not ticket queues, decide when feature work pauses in favor of reliability work.

Error budget = 1 - SLO. Exhausting the budget changes priorities by pre-agreed rule, not by escalation or argument.

Blameless postmortems are the standard learning mechanism after incidents; the goal is systemic fixes, not individual blame.

Simplicity is a stated virtue: fewer moving parts and simpler interactions mean fewer failure modes and easier reasoning.

SRE teams are staffed primarily with software engineers who also understand systems, networking, and operating systems.

Release engineering (build reproducibility, versioning, deploy automation) is treated as an SRE responsibility, not an afterthought.

Decision rule: if a service cannot be described by an SLI, define the SLI before setting any reliability target.

SLI = good events / valid events, e.g. HTTP 200 responses divided by all requests, or requests faster than 300 ms divided by all requests.

SLO = target value or range for an SLI over a window, e.g. 99.9% of requests succeed over a rolling 28 days.

Error budget = 1 - SLO. For 99.9% over 30 days the budget is 0.1%, about 43 minutes 12 seconds of allowed unavailability.

30-day budget reference values: 99% = 7h 12m; 99.9% = 43m 12s; 99.95% = 21m 36s; 99.99% = 4m 19s.

SLA is a contract with consequences (often financial); SLOs are internal targets and are usually set stricter than the SLA.

Multi-window, multi-burn-rate alerting pages on fast burn (e.g. 14.4x budget consumption over 1 hour) and slow burn (e.g. 6x over 6 hours).

Error budget policy: written pre-agreement that on exhaustion, non-essential releases freeze until reliability work restores the budget.

Aggregation matters: request-based SLIs weight by traffic volume, while time-window SLIs weight every period equally.

SLOs are set jointly with the product owner and only as tight as users actually require; tighter targets raise cost superlinearly.

Toil test (all must hold): manual, repetitive, automatable, tactical, devoid of enduring value, and scaling linearly with service growth.

Not toil: novel, one-off, or judgment-heavy work, even when it is manual and tiring; the six characteristics decide, not personal preference.

Common cap is 50% of an SRE's time on toil; teams track actual toil with time-tracking or periodic surveys rather than estimates.

Toil grows by default as a service grows; without an explicit budget it silently consumes engineering capacity.

Order of remediation: eliminate the need, then automate it, then reduce frequency or batch it - never just absorb it.

Publishing a per-team toil budget makes hidden operational load visible to management and funds automation work.

A manual step removed by automation is a permanent gain; a runbook that documents the same step is a temporary mitigation.

Decision rule: if a task recurs more than a few times per quarter and is mechanizable, automating it is usually cheaper than hiring to absorb it.

Four golden signals for user-facing systems: latency, traffic, errors, saturation.

Latency must be split into successful-request latency and failed-request latency; mixing them hides fast failures.

USE (utilization, saturation, errors) is the resource checklist; RED (rate, errors, duration) is the service checklist.

Common SLI categories: availability, latency, throughput, durability, correctness, and freshness (data staleness).

Use percentiles (p50/p90/p99), not means; tail latency drives perceived slowness, especially with fan-out requests.

Alert on symptoms that users experience (errors, latency, availability) rather than causes (high CPU) to reduce pager noise.

Black-box monitoring probes from outside as a user would; white-box monitoring reads internal metrics and logs. Use both.

Saturation is the predictive signal: it shows capacity exhaustion before users notice failure.

Dashboards should be few and high-signal; every alerting page should imply a specific, actionable response.

Automation must be idempotent so retries, crash recovery, and periodic reconciliation converge on the same state.

Ship changes by progressive rollout: canary to a small traffic slice, compare SLIs to baseline, then expand or auto-rollback.

Configuration as code: all service and infrastructure config lives in version control, is reviewed, and is reproducible.

Automation removes the manual step permanently; a runbook that walks a human through the step is a stopgap.

Automation amplifies errors, so it needs rate limits, dry-run modes, and blast-radius controls before wide deployment.

Self-service tooling lets product teams provision and deploy within guardrails instead of filing tickets to an SRE team.

The SRE toolchain typically includes CI/CD pipelines, load generators, chaos experiment frameworks, and incident management tooling.

ChatOps and incident tooling shorten coordination time by putting runbooks and status in the same channel as the responders.

Decision rule: automate the step that is repeated most often per unit of human effort first, not the most technically interesting one.

Blameless postmortem: assume people acted reasonably given available information, and attribute causes to systems and processes.

Typical postmortem triggers: user-visible incidents, data loss, SLA breach, on-call pages, and any incident someone asks to review.

Action items need an owner, priority, and tracking; unowned or unverified action items predict repeated incidents.

Chaos engineering method: define steady state, state a hypothesis, inject realistic failure, minimize blast radius, run in production.

Resilience patterns: redundancy, graceful degradation, load shedding, circuit breaking, timeouts, retries with backoff and jitter.

DiRT (Disaster Recovery Testing) rehearses large-scale failures such as loss of a datacenter or a critical dependency.

Human error is treated as a symptom of a system that permitted the error; the fix targets the system, not the operator.

Failure is unavoidable, so it is budgeted: error budgets, capacity headroom, and tested recovery paths exist before they are needed.

Anti-fragility means variability and stress produce improvement, because each failure feeds back into tested, automated remediation.

On-call should consume no more than about 25% of an SRE's time; beyond that, project work and automation stall.

Rule of thumb: more than two significant incidents in a 12-hour shift means the service or the team is overloaded and needs investment.

Error budgets align incentives: product and operations both want a healthy budget, so trade-offs become quantitative instead of political.

SRE team models: embedded (SREs join a product team), consulting (advice only), platform/infrastructure ownership, and SRE-as-a-service.

SRE does not silently absorb operations for other teams; it transfers capability so product teams own their reliability.

Blameless culture requires explicit leadership protection: people must be able to report errors without punishment.

Reliability work must be planned and resourced in the roadmap; leftover time is never sufficient.

Decision rule: when a team's toil exceeds 50%, shift the SRE engagement from feature-project work to automation and capability transfer.

"Class SRE implements interface DevOps": SRE is a concrete implementation of DevOps principles, adding SLOs, error budgets, and postmortem practice.

CAMS is the DevOps framing: Culture, Automation, Measurement, Sharing; SRE contributes most directly to Measurement.

DORA four keys: deployment frequency, lead time for change, change failure rate, and time to restore service.

ITIL relies on change advisory boards and gatekeeping; SRE replaces pre-approval with error budgets, progressive rollout, and fast rollback.

Lean contributes waste elimination (toil is waste), small batch sizes, and work-in-progress limits.

Agile supplies delivery cadence; SRE supplies the reliability constraints that cadence must respect.

Chaos engineering belongs to resilience engineering, which studies how complex socio-technical systems actually fail.

Decision rule: use SLO-based release gating where change velocity dominates; retain ITIL-style process where regulation requires auditable approvals.

SRE practices are extending to data and ML platforms, with SLIs for model freshness, training staleness, and prediction quality.

Platform engineering is the growth path: internal developer platforms make reliability defaults (SLOs, rollout, observability) self-service.

FinOps brings cost into engineering metrics, managed alongside reliability and latency rather than by a separate finance process.

Reliability is increasingly treated as a product with named owners, roadmaps, and explicit user-facing SLOs.

AI-assisted operations shift effort from incident response toward anomaly detection, capacity forecasting, and automated remediation.

Serverless and managed services move the operations boundary, changing which SLIs a team actually controls.

Smaller organizations adopt a lightweight SRE model: a handful of SLOs, one on-call rotation, and shared blameless postmortems.

Emerging and still unsettled: carbon-aware scheduling and sustainability targets as reliability-adjacent objectives.