IBM Cloud Professional SRE cheat sheet

SLI is a quantitative measure of service behavior (e.g., request latency, error rate); SLO is a target range for that SLI; SLA is a contractual commitment with consequences.

Error budget = 1 − SLO. A 99.9% availability SLO over 30 days allows about 43.2 minutes of full unavailability (0.001 × 43,200 min).

Burn rate = observed error rate ÷ (1 − SLO). A burn rate of 1 exactly exhausts the budget at the end of the SLO window; 14.4× exhausts 2% of a 30-day budget in one hour.

Toil is manual, repetitive, automatable, tactical work with no enduring value; SRE teams typically cap toil near 50% of effort and spend the remainder on engineering.

MTTR decomposes into detect, acknowledge, diagnose, repair, and verify time; reducing detection and diagnosis latency usually yields the largest reliability gains.

Availability of serial dependencies multiplies: 0.999 × 0.999 × 0.999 ≈ 0.997. Redundancy in parallel raises availability only if failures are not correlated.

Service tiers drive SLO strictness: a Tier 1 customer-facing service commonly targets 99.9%–99.99%, while internal batch tooling may target 99% or less.

Blameless postmortems record timeline, contributing factors, detection gaps, and action items with owners and due dates; the goal is systemic fix, not individual fault.

The three pillars are metrics (aggregated numeric time series), logs (discrete timestamped events), and traces (request paths across services); IBM Cloud Monitoring provides metrics and IBM Cloud Logs provides log analysis.

Four golden signals: latency (separate successful vs failed), traffic, errors, and saturation. USE method (utilization, saturation, errors) fits resources; RED method (rate, errors, duration) fits request-driven services.

Prometheus-style alerting uses alert rules over time-series expressions; alert on symptoms (SLO burn) for pages and on causes (CPU, disk) for tickets.

Multi-window multi-burn-rate alerting: page on 14.4× over 1h and 6× over 6h; ticket on 3× over 1d and 1× over 3d, reducing false pages.

Trace context uses W3C traceparent headers to propagate trace ID and span ID; sampling decisions should be recorded in the trace flags byte.

Incident severity levels (SEV1–SEV4) map to response times and communication cadence; a SEV1 typically requires an incident commander, scribe, and comms lead.

IBM Cloud Activity Tracker records account-level API audit events (who did what, when, from where) and forwards to IBM Cloud Logs or Object Storage for retention.

Dashboards should show SLO attainment and error-budget remaining first; per-pod CPU graphs belong in drill-down views, not top-level pages.

Terraform workflow: init (download providers), plan (diff desired vs state), apply (execute), destroy. State must be stored remotely with locking to prevent concurrent applies.

IBM Cloud provider for Terraform exposes resources such as ibm_resource_instance, ibm_is_instance, and ibm_container_vpc_cluster; the IBM Cloud Schematics service runs Terraform as managed workspaces.

Idempotency means re-running automation converges to the same state; scripts that append or increment are not idempotent and break repeated runs.

Immutable infrastructure replaces instances rather than patching them in place, eliminating configuration drift and making rollback a redeploy of a known image.

GitOps uses a Git repository as the single source of truth; a controller continuously reconciles cluster state to the committed manifests.

Secrets must never be committed to Git; use IBM Cloud Secrets Manager or Key Protect and inject at runtime via environment or mounted volume.

CI pipeline stages: build, unit test, static and dependency scan, image build, image vulnerability scan, sign, deploy to staging, integration test, promote to production.

Blue-green deploys switch traffic between two identical environments; canary deploys shift a small percentage of traffic first and require automated rollback on SLO breach.

Account hierarchy: an IBM Cloud account contains resource groups; access is granted via IAM policies scoped to account, resource group, or a specific resource instance.

Regions are independent geographic areas (e.g., us-south, eu-de, jp-tok); a zone is an isolated fault domain inside a region. Multi-zone deployment is the baseline for high availability.

VPC infrastructure provides subnets per zone, security groups (stateful, instance-level) and network ACLs (stateless, subnet-level); both must permit traffic for it to flow.

IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud provide managed control planes; worker nodes live in customer VPC subnets and are billed per worker.

IBM Cloud Object Storage uses buckets and supports Standard, Vault, Cold Vault, and Flex tiers; immutability and retention policies support compliance use cases.

IBM Cloud Databases (e.g., PostgreSQL, MongoDB) are managed resource instances; backups and point-in-time recovery are configured per service and should be tested by restore drills.

Direct Link provides private connectivity from on-premises to IBM Cloud; transit gateway interconnects VPCs and classic infrastructure without full-mesh peering.

Resource instances are created inside a resource group and a region; deleting a resource group deletes contained resources, so use separate groups per environment.

Design for N+1 or N+2 redundancy so a single zone or node failure does not reduce capacity below peak demand; validate with failure injection.

Capacity headroom rule: provision for peak load plus a safety margin (commonly 30%–50%) and scale on a leading indicator such as queue depth or request rate, not CPU alone.

Horizontal Pod Autoscaler scales replicas on metrics; Vertical Pod Autoscaler adjusts requests/limits; cluster autoscaler adds worker nodes when pods are unschedulable.

Little's Law: L = λ × W (concurrent requests = arrival rate × time in system). Use it to size connection pools and thread pools from latency targets.

Load test types: smoke (minimal), load (expected peak), stress (beyond peak to find breaking point), soak (hours to reveal leaks), spike (sudden burst).

Percentile latency (p95, p99) reveals tail behavior hidden by averages; set SLOs on percentiles because a mean can look healthy while 1% of users time out.

Chaos experiments should have a steady-state hypothesis, a bounded blast radius, and an automatic abort; run them in production only after staging validation.

Backup reliability requires documented RPO and RTO, restore testing at a defined cadence, and verification that restored data matches expected checksums.

IAM policy = subject (user, service ID, access group) + role + scope (account, resource group, resource). Least privilege means scoping to the narrowest resource and lowest sufficient role.

IBM Cloud IAM roles: platform roles (Viewer, Editor, Operator, Administrator) govern resource management; service roles (Reader, Writer, Manager) govern data-plane access.

Key Protect and Hyper Protect Crypto Services manage encryption keys; envelope encryption wraps data keys with a root key held in a hardware security module.

Activity Tracker events feed compliance evidence; configure a route to Object Storage with retention matching the audit requirement (often one year or more).

Vulnerability management lifecycle: inventory, scan, prioritize by exploitability and exposure, remediate within SLA by severity, then verify closure.

Compliance frameworks (SOC 2, ISO 27001, PCI DSS, HIPAA) impose different controls; map each control to an implemented, tested, and evidenced mechanism.

A service ID with an API key is the standard identity for automation; rotate keys on a schedule and store them in Secrets Manager, never in code or config files.

Security groups and network ACLs implement defense in depth; audit rules deny-all by default and allow explicit ports, protocols, and sources only.