Operations, Automation, and Governance
Operations, Automation, and Governance
This domain covers how changes reach production safely, how you know the system is healthy, and how an organization stays compliant as it scales. Items describe a team wanting more deployment autonomy or a drift incident and ask what control actually prevents the failure.
Automation as the operating model
- Infrastructure as code for everything: networks, identity, clusters, dashboards, alerts. Anything not in version control is not reproducible and will drift.
- Pipelines run plan on pull request and apply on merge, with approval gates for production. Use one pipeline for all environments; differences are parameters, not forks.
- Policy as code inside the pipeline gives fast feedback; organization policies at the platform layer are the backstop. You need both, because a pipeline can be bypassed.
Observability and operations
- Three signals: metrics are cheap and aggregatable, logs are detailed and expensive (sample them and set retention by tier), traces show causality across services. Propagate correlation IDs end to end.
- Track the golden signals per service: latency, traffic, errors, saturation.
- Maintain runbooks for known failure modes; incidents need severity levels, one incident commander, and a blameless postmortem with tracked action items.
- Small, frequent, reversible changes with automated rollback beat infrequent large releases every time.
Worked example
Thirty engineers want independent production deploys. Ungoverned, this becomes chaos. Design instead: a golden-path pipeline template (build, test, security scan, policy check, 5% canary, full rollout), org policies denying public storage and unapproved regions, mandatory tags enforced at deploy time, deploys gated on SLO burn rate, and audit logs shipped to a separate immutable project. The team gains autonomy inside guardrails, and the review burden shifts from approving individual changes to designing policies once.
Traps that cost marks
- A one-off console fix that never returns to code, creating permanent drift.
- Dashboards nobody reads and alerts on raw thresholds rather than SLO burn.
- Treating governance as manual approvals: it slows delivery and still misses the real risk.
- Missing ownership metadata, so nobody knows whom to page at 3 a.m.