IBM Cloud Associate SRE cheat sheet

SLI is a ratio of good events to valid events (e.g., HTTP 200s under 300 ms / all requests); SLO is the target value for that SLI over a defined window (e.g., 99.9% over 30 days).

Error budget = 1 − SLO. For a 30-day month (43,200 minutes): 99.9% allows 43.2 min, 99.95% allows 21.6 min, 99.99% allows 4.32 min of unavailability.

Availability = MTBF / (MTBF + MTTR); MTTR = total downtime ÷ incident count. Reducing MTTR raises availability faster than reducing incident frequency.

Error budget policy is a pre-agreed decision rule: budget exhausted → feature release freeze and reliability work takes priority; budget healthy → ship faster.

Toil is manual, repetitive, automatable, tactical work with no enduring value that scales linearly with growth; the SRE model caps toil near 50% of engineering time.

Tier SLOs by user impact: customer-facing transactional paths usually need 99.9%–99.99%; internal batch and admin tooling can sit at 99% or lower.

Alert on SLO burn rate rather than raw CPU/memory thresholds; resource alerts describe causes, budget alerts describe user-visible pain.

Blast radius is bounded by change size: canary and progressive rollout of 1% → 10% → 50% → 100% with automatic rollback on SLO regression.

Three pillars: metrics (aggregated numeric time series), logs (discrete timestamped events), traces (request path across services with span durations).

Four golden signals for request-driven services: latency, traffic, errors, saturation. USE (utilization, saturation, errors) fits resources; RED (rate, errors, duration) fits services.

Use histograms or summaries, not averages, to compute p50/p95/p99; averages hide tail latency that a small fraction of users experiences as total failure.

Multi-window burn-rate alerting for a 30-day SLO: page at 14.4× burn over 1 h (2% of budget) and 6× burn over 6 h (5% of budget); ticket at 1× over 3 days.

IBM Cloud Monitoring is Sysdig-based: platform metrics come automatically for many services, and Kubernetes clusters need an agent DaemonSet for container and app metrics.

Never put unbounded dimensions (user ID, request ID, URL with parameters) into metric labels - each unique combination creates a new time series and inflates cost.

Page only for conditions that are urgent, actionable, and require a human; everything else becomes a dashboard, ticket, or log-based alert.

A missing-data alert is required for every paging metric: absence of metrics is indistinguishable from a healthy service unless you alert on it.

Severity levels are decision rules: Sev1 = full outage or data loss (all-hands, page immediately, status updates every 15–30 min); Sev3 = degraded non-critical function handled in business hours.

Role split: Incident Commander (decides, coordinates, does not debug), Operations Lead (technical work), Communications Lead (stakeholders/status page), Scribe (timeline).

Declare the incident early - the cost of declaring is a false alarm, the cost of not declaring is an extended outage with no coordination.

Use one incident channel and one status page as single source of truth; verbal-only decisions get lost on shift handoff.

Blameless postmortem assumes people acted reasonably with the information they had; it names systemic and process causes, never individuals.

Mandatory postmortem triggers: any SLO breach exceeding budget, any Sev1/Sev2, data loss or corruption, security event, and any failed change requiring emergency rollback.

Postmortem must contain impact quantified as duration × affected users/requests × revenue, a timestamped timeline, contributing factors, and detection gap analysis.

Every action item needs one named owner and a due date; track postmortem action completion rate as a reliability metric, not as a suggestion list.

Pipeline gates in order: build → unit test → dependency and image scan → deploy staging → integration/smoke test → progressive production rollout → SLO verification.

Deployment strategies: rolling update (default, no extra capacity), blue-green (instant switchback, double cost), canary (percentage traffic, best signal-to-risk ratio).

Automated rollback must be tested in staging; deploying images by digest or immutable version tag makes rollback deterministic - latest breaks it.

Kubernetes probes: liveness failure restarts the container, readiness failure removes the pod from Service endpoints without restarting, startup probe protects slow-boot apps from liveness kills.

IaC via Terraform with remote state (IBM Cloud Schematics or CLI); always plan and review before apply, and never let two pipelines mutate the same state.

GitOps models desired state as files in Git with a controller reconciling drift; the Git history becomes the audit trail of every production change.

Automation must be idempotent and safe to retry: exponential backoff with jitter for API calls, and deduplication keys for side-effecting operations.

Measure automation coverage on the toil backlog; if a repetitive task appears three times, it becomes a script, then a pipeline step.

Account hierarchy: Account → resource groups (not nestable) → resources. IAM policies can scope to account, resource group, or a single resource instance.

IBM Cloud multizone regions expose three availability zones; a VPC spans a region while each subnet is bound to exactly one zone, so HA designs need one subnet per zone.

IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud provide an IBM-managed control plane; worker nodes run in your account (VPC or classic) and are billed per node hour.

IBM Cloud Code Engine runs containers and batch jobs serverless with scale to zero, removing cluster and node management for suitable workloads.

Object Storage (COS) storage classes - Standard, Vault, Cold Vault - trade retrieval latency for cost; resiliency options are cross-region, regional, and single-site.

Hybrid workloads commonly combine IKS/OpenShift with managed databases, Key Protect or Secrets Manager, COS, and a VPC load balancer or CIS at the edge.

Key management split: Key Protect manages root keys for envelope encryption, Secrets Manager stores rotating application credentials; Hyper Protect Crypto Services provides the highest FIPS assurance level.

Tag resources consistently (owner, environment, cost center) - tags drive cost allocation and are the fastest way to find unowned prod capacity.

VPC security groups are stateful and attach to instances (return traffic auto-allowed); network ACLs are stateless and attach to subnets, evaluated top-down, requiring explicit rules in both directions.

Application load balancers handle HTTP/HTTPS with TLS termination and path routing; network load balancers handle L4 TCP/UDP where the protocol is not HTTP or source IP preservation matters.

IBM Cloud Internet Services (CIS) provides global DNS, CDN, WAF, DDoS mitigation, and TLS at the edge; proxy mode hides origin IPs but must be disabled for non-HTTP protocols.

Transit Gateway is the private routing hub connecting VPCs, classic infrastructure, and Direct Link attachments across accounts and regions without traversing the public internet.

Direct Link gives private, predictable-latency connectivity to on-premises; keep a site-to-site VPC VPN as a documented failover path and test the failover.

DNS TTL governs failover speed - lower TTLs (60 s or less) well before a planned migration, and remember caches honour TTL after you change a record.

Public gateways provide per-zone outbound internet for VPC subnets; inbound internet traffic requires a floating IP or a load balancer, not a public gateway.

VPC flow logs capture connection metadata (source, destination, bytes, action) into COS or log destinations; they are the primary tool for proving whether traffic was allowed.

IBM Cloud IAM subjects: users, service IDs, and API keys. Group them into access groups and attach policies to the group so onboarding and offboarding change membership only.

Two role families: platform management roles (Viewer, Editor, Operator, Administrator) and service access roles (Reader, Writer, Manager plus service-specific roles); a policy needs both to be useful.

API keys are displayed once at creation and cannot be retrieved later; store them in Secrets Manager, rotate on a schedule, and never bake them into images or Git.

Least privilege rule: grant Administrator at account level only for the few identities that manage IAM itself; everything else scopes to a resource group or resource instance.

Activity Tracker Event Routing records account-level API, CLI, and console events with outcomes - it is the audit evidence for who did what and when, not a metrics system.

Envelope encryption: data is encrypted with a data encryption key, and that key is wrapped by a root key in Key Protect; revoking the root key renders the data unreadable.

Shared responsibility: IBM secures the physical, hypervisor, and managed-service layers while the customer owns identity configuration, network rules, encryption choices, and data classification.

Encryption defaults: TLS 1.2+ in transit for IBM Cloud endpoints, encryption at rest for VPC block storage and COS; use virtual private endpoints so service traffic never leaves the IBM backbone.

Rightsize from p95 utilization over a full weekly cycle, then keep headroom for the loss of one zone (N+1) rather than sizing only to average load.

Kubernetes requests drive scheduling and QoS (Guaranteed when requests equal limits, Burstable otherwise, BestEffort with none); limits above requests risk OOMKill and CPU throttling.

Horizontal Pod Autoscaler scales replicas on CPU, memory, or custom metrics and depends on a working metrics pipeline; cluster autoscaler then adds worker nodes when pods stay Pending.

Troubleshooting order: confirm the symptom against the SLO → check recent changes → metrics → logs → traces → infrastructure, stopping at the first layer that explains the data.

Pod state decoder: Pending = unschedulable resources/taints; ImagePullBackOff = bad image reference or missing pull secret; CrashLoopBackOff = app or probe failure; OOMKilled = memory limit exceeded.

Latency triage splits client time, network round-trip (including DNS resolution), and server processing time; traces localize which segment grew before you touch code.

Load test at the traffic level you expect at peak plus one zone down; watch saturation signals - queue depth, connection pool exhaustion, thread counts - not just CPU.

Store capacity evidence (test results, scaling events, cost of headroom) with the service so right-sizing decisions are reviewable rather than folklore.