SRE Practitioner cheat sheet

Toil is manual, repetitive, automatable, tactical work with no enduring value that scales linearly with service growth; common guidance caps total operational load, including toil, at 50% of an SRE team's time.

Blameless postmortems assume responders acted reasonably given the information they had, so analysis targets systemic and process conditions rather than individual punishment.

Every postmortem action item carries a named owner and a due date; an item without an owner is treated as unfinished work, and recurring contributing factors signal a process failure.

Change is a leading cause of outages (Google has reported roughly 70%), so the cultural response is progressive rollout, canarying, and fast automated rollback rather than heavier change-approval boards.

On-call load target: about two paging events per 12-hour shift so responders retain time to engineer; paging above that level is treated as a defect in alerting or reliability, not an accepted cost.

Error budgets align incentives: because releases consume budget, product and SRE teams negotiate the same velocity-versus-reliability trade-off with shared data instead of escalations.

A production readiness review gates launch on concrete evidence: SLIs and SLOs defined, dashboards live, runbooks tested, capacity plan, and a proven rollback path.

Shared ownership means the team that builds a service carries its pagers; SRE acting as a permanent human shield for another team's service is a modelling failure, not a service.

Antipattern: renaming an operations team 'SRE' while leaving on-call structure, toil levels, and release authority unchanged; the label changes, the outcomes do not.

Antipattern: a 100% SLO target. It leaves no error budget, blocks every release, and is unattainable in a distributed system where independent components fail independently.

Antipattern: paging on every symptom or on raw resource thresholds. Page on user-facing SLO burn rate; route resource saturation signals to tickets or dashboards.

Antipattern: averages for latency. A 200 ms mean can conceal a p99 above 5 seconds; use percentiles and per-user-journey SLIs.

Antipattern: hero culture, where recovery depends on one person's memory. The corrective pattern is automation plus documented, rehearsed runbooks.

Decision rule: if a task is repeatable and machine-executable, it is an automation candidate; if it is only documented, the documentation is a stopgap, not the strategy.

Error budget policy tiers: budget exhausted → feature freeze with reliability work prioritized; budget nearly exhausted → added release scrutiny and a named approver.

Design principle: assume components will fail and engineer graceful degradation and defense in depth, rather than optimizing for the prevention of all failures.

Antipattern: an SLO treated as a vanity metric. Every SLO must map to at least one alerting decision and one release decision, or it is decoration.

Adoption typically starts with a small SRE team owning a few services and spreads by demonstrated results; mandate-driven, organization-wide rollouts usually stall.

Structural choices include embedded SRE (co-located with product teams), centralized SRE (a shared service team), and enabling/consulting SRE; each trades adoption speed against depth of ownership.

SRE is a software engineering role: hiring profiles and time budgets must reserve roughly half of capacity for engineering work, or the practice degenerates into operations.

Error budget policies hold only when an executive sponsor enforces the release freeze; without that authority, the budget is advisory and is overridden during the first roadmap conflict.

Handover sequence matters: product teams take primary on-call before SRE assumes responsibility, otherwise SRE-on-call for a service without dev on-call recreates the human shield.

Leading indicators of a healthy practice: paging events per shift trending down, engineering-time share trending up, time-to-mitigate shrinking, and the share of services with enforced SLOs rising.

Introducing error budgets moves decision rights from operations to a negotiated reliability target, which predictably produces turf conflict with product management and must be planned for.

Psychological safety is a prerequisite for blameless postmortems; leaders establish it by how they behave in the first few incidents, not by writing it into a policy document.

SLI = good events / valid events, with both numerator and denominator defined precisely (for example, HTTP 2xx and 3xx responses over all non-health-check requests).

Error budget = 1 − SLO. A 99.9% availability SLO over 28 days permits about 40 minutes of full unavailability, or proportionally more at partial failure.

Burn rate = observed error rate / (1 − SLO). A burn rate of 1.0 consumes the entire budget over exactly one SLO window.

Multiwindow, multi-burn-rate alerting: page at 14.4× burn over 1 hour (2% of a 30-day budget) and 6× over 6 hours (5%); use slower burn rates for ticket-level alerts.

Match the window to the user: rolling 28 days for a stable target, shorter windows for fast-moving services; avoid calendar-aligned windows because they reset budgets unpredictably.

SLI categories: availability, latency (fraction of requests faster than a threshold), throughput, freshness, correctness, and durability; pick the one that reflects user pain.

In a distributed system measure SLIs both at service boundaries and end to end on the user journey; a healthy per-service SLI can still hide a broken composite journey.

Never average ratios across services. Aggregate by summing good events and valid events, then divide; averaging percentages weights a low-traffic service the same as a high-traffic one.

Each SLO needs an owner, a dashboard, an alerting rule, and a documented action taken when the budget is exhausted.

Zero trust grants no implicit trust from network location: every request is authenticated and authorized on its own merits, following least privilege.

Workload identity via mTLS (SPIFFE/SPIRE identities, short-lived certificates) replaces perimeter trust; certificates rotate automatically and revocation is part of the design.

Assume breach: default-deny network policy, microsegmentation, and per-service authorization limit how far a single compromised component can move.

Resilience and security share mechanisms: cells, zones, and bulkheads that contain failures also contain intrusions, so blast-radius design is a joint investment.

Human access uses just-in-time elevation instead of standing admin; break-glass accounts exist, are used rarely, and generate alerts and audit records on use.

Security SLI examples: percentage of requests mutually authenticated, percentage of critical patches applied within SLA, and time from CVE disclosure to remediation.

Supply chain controls: signed artifacts, SBOMs, and build provenance (SLSA) make a compromised dependency detectable and blockable at deploy time rather than after exploitation.

Chaos experiments can validate security controls (revoke a credential, verify denial), but must be scoped and approved so they never amount to real privilege abuse.

Pillars are metrics, logs, and traces, extended by profiles and events; correlation keys such as trace ID and request ID are what tie the signals into one narrative.

Trace context propagates through W3C Trace Context headers (traceparent, tracestate); asynchronous hops through queues must explicitly inject and extract context or the trace silently breaks.

A trace is a tree of spans; each span records service, operation, duration, status, and attributes, and parent linkage reconstructs the request path.

Head-based sampling decides at request start (cheap, blind to outcome); tail-based sampling decides after completion (retains errors and slow traces, requires buffering in a collector).

Exemplars attach trace IDs to metric samples, so a latency spike on a dashboard links directly to a representative trace.

RED (rate, errors, duration) covers request-driven services; USE (utilization, saturation, errors) covers resources such as CPU, memory, and disks.

High-cardinality labels such as user ID or request ID on metrics cause storage and cost blowups; keep them in traces and logs, not in metric label sets.

Observability-driven development means instrumenting while writing code, asserting on telemetry in automated tests, and using traces in CI to catch regressions before production.

OpenTelemetry supplies a vendor-neutral API, SDK, and collector pipeline, decoupling instrumentation in the code from the backend that stores the data.

AIOps applies machine learning to telemetry for anomaly detection, alert correlation, noise reduction, root-cause suggestion, and forecasting; it augments responder judgment rather than replacing it.

Baseline choice drives outcome quality: static thresholds generate toil and miss seasonality, while dynamic baselines need sufficient history and must themselves be monitored for drift.

Alert correlation groups alerts by topology, time, and shared entity so that one outage becomes one incident instead of hundreds of pages.

Supervised models require labeled incident outcomes; without a feedback loop that labels alerts with what actually happened, model precision decays over time.

Predictive use cases with practical payoff: capacity forecasting, disk failure prediction from SMART telemetry, and projected error-budget exhaustion from current burn.

DataOps supplies the foundation: versioned, tested, automated data pipelines with schema validation and lineage, so models and dashboards are reproducible.

Guardrails: human-in-the-loop confirmation, explainable root-cause suggestions, drift monitoring, and the ability to fall back to deterministic rules.

Telemetry governance: redact PII at the collector, retain high-fidelity data briefly and aggregates longer, and record provenance for any data used in training.

An internal developer platform offers self-service through golden paths; its purpose is to reduce cognitive load, not to insert a new approval gate.

Platform as a product: developers are the customers, and success is measured by adoption, time-to-first-deploy, change failure rate, and support-ticket volume.

Golden paths are secure-by-default templates with CI/CD, observability, and SLO scaffolding pre-wired; leaving the path is allowed but requires explicit justification.

Guardrails over gates: policy-as-code (for example OPA or Gatekeeper) blocks non-compliant deployments automatically instead of relying on review meetings.

Consistency removes whole classes of incidents: one deployment pipeline, one telemetry schema, and one secret-management pattern across all services.

In Team Topologies terms a platform team provides X-as-a-Service and enabling support; it should not become a ticket queue that every change has to pass through.

Bake reliability into the pipeline itself: automated SLO checks, progressive delivery, and error-budget-based release gating rather than manual sign-off.

Platform services need their own SLOs, and stricter ones than their tenants: a platform outage is simultaneously an outage for every team it serves.

Chaos engineering is the disciplined use of experiments on a system to build confidence in its ability to withstand turbulent conditions, ideally including production.

Standard loop: define the steady state, hypothesize it holds, inject a real-world variable, try to disprove the hypothesis, then fix and automate the experiment.

Steady state must be measurable as an SLI range or similar signal; 'the system looks normal' is not a steady-state definition and yields no verdict.

Blast-radius controls: run in staging first, then a canary, then a small production percentage; define abort conditions and a kill switch before the experiment starts.

Common experiments: instance or pod termination, network latency and packet loss, dependency or API failure, availability-zone failure, disk fill, and clock skew.

Prerequisites are SLOs and observability to detect impact, automated rollback to recover, and an informed on-call rotation for the duration of the experiment.

Game days rehearse the human response alongside the technical failure: detection, escalation, decision-making, and communication, inside a planned window.

Chaos in CI: automated, low-blast-radius experiments in staging or ephemeral environments catch resilience regressions on every pull request.

Core roles: Incident Commander (coordinates and decides, does not debug), Operations Lead, Communications Lead, and Scribe; on small incidents one person may hold several roles.

The Incident Commander sets severity, delegates work, maintains the timeline, and holds sole authority over mitigation trade-offs during the incident.

Mitigate first, diagnose later: stopping customer impact (rollback, failover, feature flag disable) takes priority over determining root cause.

Severity is classified by customer impact and scope, and it determines who is paged, the update cadence, and whether executives or customers are notified.

Communication cadence: scheduled stakeholder updates roughly every 15 to 30 minutes, plus a status page maintained by the Communications Lead rather than by the Incident Commander.

Time to mitigate is the headline metric; reporting only MTTR conflates detection lag, mitigation time, and permanent repair in a single number that hides the actionable part.

Long incidents require explicit handoffs for the Incident Commander and Operations Lead, with a written summary of current state, active hypotheses, and next actions.

The blameless postmortem follows within days and produces tracked action items; recurrence of the same contributing factor indicates a process failure, not bad luck.

Escalation paths and paging policies must be tested in advance; a page that nobody acknowledges is a paging-configuration defect that should be fixed immediately.