DevOps Engineering cheat sheet

Pipeline stage order that satisfies both speed and safety: lint/static analysis → unit tests → build artifact once → integration tests → security scan → deploy to staging → smoke tests → progressive production rollout.

Build once, promote many: the same immutable artifact digest must flow from staging to production; rebuilding per environment breaks provenance and reproducibility.

A pipeline is only as reproducible as its runner image; pin CI runner and tool versions by digest, not by floating tags such as latest.

Fail fast ordering rule: place the cheapest, highest-signal checks first so median feedback time stays low; a 10-minute unit suite before a 30-second linter wastes developer time.

Blue-green deployment switches traffic atomically between two identical environments and enables instant rollback by reverting the router; canary releases shift a percentage of traffic and require automated metric gates.

Deployment frequency, lead time for changes, change failure rate, and time to restore are the four DORA delivery metrics; they are measured from pipeline and incident data, not surveys.

Idempotent pipeline steps can be safely retried; non-idempotent steps (for example, a migration that inserts rows) need explicit guards or compensating actions before retry logic is added.

Secrets belong in a managed secret store injected at runtime, never in pipeline YAML or build logs; mask variables and restrict who can trigger pipelines that expose them.

Artifact retention and cache invalidation rules must be explicit: caches keyed on lockfile hashes avoid stale dependency reuse, and retention windows bound storage cost and supply-chain exposure.

Declarative IaC describes desired end state and lets the tool compute the diff; imperative scripts describe steps and are not self-healing after drift.

Terraform state is the mapping between configuration and real resources; store it remotely with locking (for example, object storage plus a lock table) to prevent concurrent-apply corruption.

terraform plan is a dry run showing proposed changes; review the plan in pull requests and treat any unexpected destroy as a blocker.

Idempotence is the core property of configuration management: applying the same manifest twice must produce no changes on the second run.

Puppet and Chef are agent-based and pull configuration on a schedule; Ansible is typically agentless over SSH and push-based; choose based on existing fleet and network constraints.

Immutable infrastructure replaces servers rather than mutating them; mutable configuration drift is the main reason 'works on my machine' recurs in production.

Modules and roles exist to remove duplication: a Terraform module should expose inputs with validation and outputs, and avoid hard-coded provider or region values.

Drift detection compares live infrastructure against the declared state on a schedule; unmanaged drift is a leading indicator of incident risk and audit findings.

Shared responsibility model: the provider secures the cloud (physical, hypervisor, managed service internals); the customer secures what runs in the cloud (identity, data, configuration, network policy).

Least privilege is enforced through identity policies and roles; long-lived access keys should be replaced by short-lived credentials from instance, workload, or OIDC federation.

Availability zones are isolated datacenters within a region; designing across at least two AZs removes single-datacenter failure as a cause of outage.

Autoscaling groups or managed node pools should scale on a signal correlated with load (queue depth, request rate) rather than CPU alone, which lags real demand.

Managed services trade operational burden for reduced control; the decision rule is whether the team's differentiation lies in running that service or in the product it supports.

Cost control levers: tagging for allocation, budgets and anomaly alerts, right-sizing from utilization data, and lifecycle policies on object storage.

Infrastructure quotas and service limits are per-region and per-account; request increases before load events, because hitting a quota looks like an application failure.

Multi-region designs must address data replication lag and conflict resolution; active-active requires a strategy for writes, not just for reads.

A container image is an immutable, layered filesystem plus metadata; layers are content-addressed and cached, so ordering Dockerfile instructions from least to most frequently changing maximizes cache reuse.

Multi-stage builds separate build toolchains from runtime images, shrinking attack surface and image size; the final stage should contain only runtime dependencies.

Kubernetes liveness probes restart unhealthy containers; readiness probes remove pods from service endpoints; using a liveness probe for dependency checks causes cascading restarts.

resource requests determine scheduling and guaranteed capacity; limits cap usage and cause CPU throttling or OOM kills when exceeded. Set requests from observed usage.

A Pod is the smallest schedulable unit and shares a network namespace; containers in the same Pod reach each other over localhost.

Deployments provide rolling updates with maxSurge and maxUnavailable controls; rollback is a revision change, so keep revision history depth sufficient for the rollback window.

Horizontal Pod Autoscaler scales replicas from metrics; cluster autoscaler or node provisioning must be able to add capacity, or pods stay Pending.

Namespaces provide scoping and quota boundaries but are not a security boundary by themselves; network policies and RBAC are required for isolation.

The three pillars: metrics are cheap numeric aggregates over time, logs are discrete timestamped events, traces follow a request across service boundaries via propagated context.

SLI is a measured ratio (successful requests / valid requests); SLO is a target for that SLI; the error budget is 1 minus the SLO, expressed over the compliance window.

Burn-rate alerting fires when the error budget is consumed faster than the window allows; a 1-hour window at 14.4x burn rate approximates a fast, high-confidence page.

Four golden signals: latency, traffic, errors, and saturation. Latency must distinguish successful from failed request durations, since fast failures skew averages.

Percentiles, not means, describe user experience; p99 latency hides nothing about the slowest users, and aggregated averages mask tail behavior entirely.

Alert on symptoms users feel (error rate, latency) rather than causes (CPU high); cause-based paging produces noise and alert fatigue.

Runbooks turn alerts into actions: each paging alert should link to a runbook with diagnosis steps, owner, and escalation path.

Postmortems are blameless and action-oriented: timeline, contributing factors, and tracked remediation items with owners and due dates.

Shift left means security checks run in the pipeline: dependency and container scanning on every build, IaC scanning on every plan, secret scanning on every push.

SAST analyzes source code for vulnerabilities; DAST probes a running application from outside; SCA checks third-party dependencies against known vulnerability databases.

A vulnerability gate needs a severity threshold and an exception process with expiry; gates with no exceptions get disabled by developers under deadline pressure.

SBOM (for example, SPDX or CycloneDX) enumerates components and versions in an artifact, enabling rapid impact analysis when a new CVE appears.

Signing and provenance attestation (for example, Sigstore/cosign and SLSA levels) let admission controllers reject unsigned or unverified images.

Policy as code (for example, OPA/Rego or admission policies) enforces guardrails automatically; manual review does not scale with deployment frequency.

Compliance evidence should be generated from automation artifacts (scan reports, policy evaluations, access reviews) rather than assembled manually at audit time.

Rotate and scope credentials continuously; a leaked key's blast radius is determined by its permissions, so least privilege limits incident severity.

Trunk-based development merges small changes to a shared mainline at least daily and hides incomplete work behind feature flags; long-lived branches increase merge conflict cost superlinearly with branch age.

Git Flow uses long-lived develop, release, and hotfix branches and suits versioned, scheduled releases; it adds merge overhead that conflicts with continuous deployment.

GitHub Flow is short-lived feature branches plus pull requests into main; it is the common default for continuously deployed web services.

Semantic versioning: MAJOR for incompatible changes, MINOR for backward-compatible features, PATCH for backward-compatible fixes; pre-release identifiers sort before the release.

A release is a deployable artifact plus metadata; a deployment is the act of putting it in an environment. Separating the two enables decoupled release and deploy cadence.

Merge versus rebase: rebase rewrites commits for linear history and must not be applied to shared branches; merge preserves history and is safe on public branches.

Tags should be immutable and signed; moving a release tag breaks provenance and reproducibility for anyone who consumed it.

Feature flags decouple deploy from release but accrue debt; each flag needs an owner and a removal date, or flag combinations become untestable.

Shebang and strict mode: #!/usr/bin/env bash plus set -euo pipefail makes scripts fail on errors, unset variables, and pipeline failures instead of continuing silently.

Quote variable expansions ("$var") in shell to prevent word splitting and globbing; unquoted expansions are a common source of destructive bugs.

Exit codes: 0 means success and any nonzero means failure; automation should branch on exit status, not on parsing human-readable output.

Idempotent scripts check current state before acting (mkdir -p, create-if-absent) so reruns converge rather than duplicate or error.

Prefer structured data formats (JSON, YAML) and parsers (jq, yq) over text scraping with awk and sed for tool output that may change format.

Python virtual environments or pinned lockfiles isolate dependencies; unpinned installs make automation non-reproducible across machines and dates.

Log to stderr and emit machine-readable results to stdout so scripts compose cleanly in pipelines and can be consumed by other tools.

Dry-run and plan modes should be implemented for destructive automation, with an explicit confirmation flag required for real execution.

The Westrum organizational typology classifies culture as pathological (power-oriented), bureaucratic (rule-oriented), or generative (performance-oriented); generative cultures predict higher software delivery performance.

Blameless postmortems assume people acted reasonably given the information available; the goal is system improvement, and punishment suppresses reporting.

Conway's Law: system architecture mirrors organizational communication structure; team topologies should be designed deliberately rather than inherited by accident.

Small batch sizes reduce lead time and risk; a change touching one service and a few hundred lines is easier to review, test, and roll back than a quarterly release.

Definition of done should include automated tests, observability, and rollback capability, not just merged code, or quality debt accumulates invisibly.

Code review turnaround is a delivery constraint: review latency counts in lead time for changes, so review SLAs directly affect the DORA metric.

Psychological safety, measured as willingness to report mistakes and dissent, is a precondition for incident learning and for effective blameless retrospectives.

Cross-functional teams owning build and run (you build it, you run it) shorten feedback loops between code and production behavior compared with handoffs to a separate operations group.