Vantage Logistics: unwinding a two-year Terraform drift problem in two sprints

The risk is not the drift; it is the next apply. Stop that first, then reconcile, then restructure. Refactoring first is how the two Cloud SQL instances die.

Sprint 1 - contain and observe.

1. Day 1: revoke laptop applies. Remove the write capability on the state bucket from human identities; leave read. Any remaining apply goes through one designated applier on the platform team. This is a policy change, not a tool.
2. Day 1-2: export the two Cloud SQL instances to a GCS bucket with object retention, and take an on-demand backup. prevent_destroy is a guardrail against Terraform; the export is the insurance against everything else, including a bad state rm.
3. Add lifecycle { prevent_destroy = true } to every stateful resource - databases, buckets, persistent disks, the VPC - and get that merged. Now the worst-case plan fails loudly instead of succeeding.
4. Produce a drift inventory: a scheduled terraform plan -detailed-exitcode in CI against each root module, and the plan JSON parsed into a table of resource → planned action → owner. 41 console changes map onto specific resources; the two Cloud SQL instances that would be destroyed should show as a replace/destroy in this output and must be the first thing triaged.
5. Triage each of the 41 into keep or revert. Keep → write the HCL and terraform import. Revert → change the console back by hand now, so the code and reality agree before any apply. Do not attempt a bulk import; a wrong import on a database is unrecoverable in a way a wrong revert is not.

Sprint 2 - restructure on a stable base. 6. Split state by blast radius, environment first, then service. Use terraform state mv within a moved block where possible; state rm + import where not, and re-run the drift inventory after each move to prove nothing changed. One root module and one state bucket folder per environment per service, with a separate project per environment. 7. Prefer directories over workspaces. Workspaces share one backend and one set of credentials, which is exactly how someone applies prod while intending staging. Directories plus per-environment service accounts make the mistake structurally hard. 8. Move state to a GCS backend with object versioning, retention, and uniform bucket-level access. Free, and it converts future state mistakes from incidents into restores.

What I would tell leadership: the two sprint limit is enough for containment and the state split, not for full GitOps. Containment is what stops the incident; the rest is cadence.

Invert the trust model: the guardrail is policy, not review. Four engineers cannot review 190 developers' pull requests, and they will stop trying within a month.

Layers, cheapest first:

1. Org Policy as the hard boundary. Deny public bucket IAM, deny external IP on VMs outside an approved folder, restrict allowed locations to the company's regions, restrict allowed APIs. These hold even when a developer writes a terrible module, and they cost nothing. This is the layer that lets you say yes to self-service.
2. A small module catalog, not a platform. 5-8 vetted modules - service skeleton, Cloud Run service, GCS bucket, Cloud SQL with standard flags, Pub/Sub topic - in a private registry or a Git repo with tags. Developers compose approved modules; the modules encode the org's opinions once instead of in 190 reviews.
3. Delegated IAM by group, per environment. Developers get deploy rights on their team's dev and staging folders via Google Groups. Production applies run from a pipeline whose service account the developer cannot impersonate; they open a change request instead. Production stays gated, and that is the one gate worth keeping.
4. Policy-as-code in the pipeline, since the license is already bought. Every plan is evaluated against the policy set; violations block the pipeline automatically. This is what replaces the human review you cannot staff.
5. A managed service catalog only if the above is not enough. Config Controller on an existing GKE cluster gives namespaced self-service resources with GitOps and RBAC for free, but it is a platform you now operate. With 4 people and 600 resources, I would not stand it up in the first quarter; the module catalog plus org policy gets 80% of the benefit for a fraction of the operational load.

What I would refuse: an approval queue for every non-production change. The reason 41 console changes happened is that the compliant path was slower than the console. Make the compliant path a 10-minute pipeline run and drift stops being a policy problem.

Reconcile one resource at a time against a written rule, and detect drift on a schedule rather than on the next apply.

Reconciliation rule, applied per change:

• Is the console change desirable? Then it is a code change: write the HCL, terraform import, review, apply. The console is never the system of record, even when it is correct.
• Is it undesirable? Revert in the console first, verify, then the code and reality agree and the next plan is empty. Reverting by editing code would mean an apply that touches production state to fix a production change - more risk for the same outcome.
• Unsure? It stays flagged and the resource gets prevent_destroy and an owner. Never bulk-import to make the plan clean; a clean plan that hides two destroys is worse than a noisy plan.

Recurrence detection, three independent mechanisms:

1. Scheduled drift detection. Hourly terraform plan in CI for production, daily for lower environments, alerting on a non-empty plan. Route the alert to the owning team with the resource name in it - a generic 'drift detected' alert gets muted in a week.
2. Audit log alerts on the mutations themselves. Admin Activity log-based alerts on compute.firewalls.create, compute.instances.setMetadata, IAM policy changes, and database flag changes, in the production projects. This catches the change in the first minute, not the next morning, and it names the principal.
3. Cloud Asset Inventory for an independent resource inventory diffed against what code claims to own. This is the only mechanism that catches a resource created entirely outside Terraform, which a plan against existing state will never show.

The structural fix, without which all three alerting mechanisms just generate noise: a documented emergency change path - a hotfix/ branch, a 15-minute pre-approved pipeline, and a 24-hour back-port requirement tracked by a label on the ticket. If the emergency path is slow, people use the console, and you will be reconciling 41 more changes next quarter.