GCP DevOps Engineer cheat sheet
GCP DevOps Engineer · Keep the key ideas within reach.
Bootstrap and maintain a Google Cloud organization
Hierarchy is Organization → Folder → Project → Resource; allow policies are additive downward, so a child project can never remove access granted at an ancestor node.
An organization resource exists only when the projects sit under a Google Workspace or Cloud Identity domain; Free Tier and consumer accounts have no organization and therefore no folders.
Folder nesting is limited to 10 levels below the organization, which caps how deep environment/team hierarchies can go.
Org Policy boolean constraints (for example constraints/iam.disableServiceAccountKeyCreation) take effect at the node where they are enforced and cannot be relaxed by descendants.
Shared VPC: the host project owns subnets, routes, and firewall rules; service projects attach to consume them; compute.xpnAdmin must be granted at the organization or folder level.
Aggregated sinks declared at folder or organization level capture logs from every descendant project into one central bucket or BigQuery dataset for audit and retention.
The _Required log bucket holds Admin Activity and System Event audit logs with 400-day retention; the _Default bucket holds other logs for 30 days by default.
Project ID is globally unique and immutable, project number is Google-assigned, and the display name is mutable; API enablement and quotas are scoped per project.
Terraform bootstrap pattern: remote state in a GCS bucket with object versioning, one state/workspace per environment, and org- and folder-level IAM resources declared before the projects themselves.
Apply site reliability engineering practices
SLI = good events ÷ valid events; the SLO is the target fraction over a window; the SLA is the contractual threshold and is normally looser than the internal SLO.
Error budget = 1 − SLO: 99.9% over 30 days is about 43 minutes; 99.95% about 21.6 minutes; 99% about 7.2 hours.
Burn rate = observed error rate ÷ (1 − SLO); multiwindow alerting uses 14.4× over 1 h (2% of budget), 6× over 6 h (5%), and 1× over 3 days (10%).
Alert on symptoms (budget burn, error ratio, latency) rather than causes (CPU, memory); cause metrics should open tickets, not page a human.
Golden signals per user-facing service: latency, traffic, errors, saturation - the minimum metric set for any SLI review.
Error budget policy: when the budget is exhausted, feature releases freeze and reliability work takes priority until the budget recovers.
Progressive delivery: canary sends a small traffic share and promotes only if SLIs hold; blue/green shifts traffic while keeping the old revision for instant rollback.
Capacity: provision N+2 for critical services, validate with load tests, and confirm failover paths with DiRT or chaos exercises against the SLO.
Postmortems are blameless, are triggered by SLO violations or abnormal burn, and produce owned, tracked action items with due dates.
Build and implement CI/CD pipelines, including continuous testing, for application, infrastructure, and machine learning workloads
Cloud Build steps run sequentially by default; waitFor: ['-'] makes a step run in parallel; the default step timeout is 10 minutes and the maximum build timeout is 24 hours.
Triggers fire on push, tag, or pull request events; substitutions such as $COMMIT_SHA and $_ENV inject per-build values; private pools give the build access to private VPC resources.
Artifact Registry stores container images and language packages with per-repository IAM, regional location, and vulnerability scanning; it replaces Container Registry for new projects.
Binary Authorization admits only images with attestations from approved attestors, blocking unsigned images at deploy time on GKE and Cloud Run.
Cloud Deploy models delivery pipeline → targets → releases → rollouts, supporting canary and linear strategies plus a verify phase and rollback to a previous release.
Terraform CI: run plan on pull request and apply on merge, store state in GCS with locking, and use a per-environment deploy service account with least privilege.
Continuous testing order: unit → integration → container/security scan → performance; a failed policy check must fail the build, not be deferred to post-deploy.
GitOps with Config Sync or Argo CD makes the Git repository the source of truth and continuously reconciles clusters, so drift is detected and corrected automatically.
ML CI/CD: Vertex AI Pipelines orchestrates data prep, training, and evaluation; a metric threshold gate decides whether the model is registered and deployed to an endpoint with a traffic split.
Implement observability practices and troubleshoot issues
Cloud Monitoring covers metrics, dashboards, uptime checks, SLOs, and alerting; Cloud Logging covers buckets, sinks, and log-based metrics; Cloud Trace and Profiler cover latency and resource hotspots.
The log router applies inclusion filters, exclusion filters, and sinks; exclusion filters drop high-volume debug entries before ingestion and are the main lever for logging cost.
Log-based metrics turn a filtered stream of entries into a counter or distribution that can drive dashboards and alerting policies without code changes.
An alerting policy is condition + notification channels + documentation; the incident auto-close duration prevents flapping alerts from opening duplicate incidents.
The Ops Agent is the single collector for VM metrics and logs; GKE uses system and workload logging plus Managed Service for Prometheus when PromQL is required.
Trace spans show which hop dominates end-to-end latency; Profiler attributes CPU and heap consumption to specific source lines and functions.
Debugging 5xx: read the load balancer log statusDetails first, then filter service logs by severity>=ERROR and the trace ID to correlate the failing request.
Uptime checks probe from multiple regions; a check failing in every region points to DNS, certificate, or global frontend faults rather than a single-zone outage.
Optimize performance and cost
Compute Engine: sustained use discounts are automatic, 1- or 3-year committed use discounts reach about 57% on a predictable baseline, and Spot VMs cut up to 91% for fault-tolerant work.
Active Assist rightsizing and idle-resource recommendations identify oversized VMs, unattached persistent disks, and unused static IP addresses.
GKE scaling: HPA scales pods on metrics, VPA right-sizes requests, cluster autoscaler adds or removes nodes, node auto-provisioning creates pools shaped to pending pods, and Autopilot bills per pod request.
Cloud Storage classes: Standard, Nearline (30-day minimum), Coldline (90-day), Archive (365-day); Object Lifecycle rules and Autoclass move objects automatically, and early deletion is charged prorated.
BigQuery cost control: partition and cluster tables to prune scanned bytes; on-demand bills per TiB scanned while editions and reservations bill per slot-hour.
Cloud Run: raise concurrency to reduce instance count, keep minimum instances only on latency-critical paths, and use request-only CPU allocation for bursty services.
Cloud CDN and Media CDN cut origin egress and latency; the Premium network tier lowers latency at a higher price than the Standard tier.
Cost governance: labels plus billing export to BigQuery enable per-team chargeback; budgets with Pub/Sub alerts notify but never cap spend, so quota limits enforce hard ceilings.