GISI ADO Level: terraform plan Wants to Delete the Production Database

terraform plan Wants to Delete the Production Database

Corvus Logistics runs a three-account AWS estate (dev, staging, prod) from one Terraform repository. State lives in S3 with DynamoDB locking, one workspace per environment, and environment differences are meant to live in .tfvars files.

During a P1 incident, an on-call engineer changed an RDS instance class from db.r6g.2xlarge to db.r6g.4xlarge in the console, added an inbound rule to a security group to unblock replication, and created an S3 bucket for log exports. None of it was codified. Engineers hold console admin in prod because "the incident needs it".

The subnet resources are declared with count over a list of six subnets. Last release, someone removed the third entry from that list; the next terraform plan against prod shows updates to the remaining subnets and destroy/create of three aws_subnet resources - and a destroy of the RDS instance, because its identifier is derived from the subnet index. There is no prevent_destroy in the code, plan output is reviewed by whoever is free at the time, and the scheduled drift job was disabled two months ago when it paged someone.

1. Production is drifting and the pending plan destroys an RDS instance. Describe your order of operations to reach a safe, codified state, and justify why you choose import, apply or config revert at each step.

Read the worked answer

Do not run terraform apply on the current plan. The plan conflates two independent problems - manual drift and an accidental config-induced re-indexing - and applying it would execute the destructive half.

1. Stop the bleeding. Save the plan to a file (terraform plan -out=tfplan) for forensics, freeze merges to the prod workspace, and turn the drift job back on in report-only mode.

2. Neutralise the destroy first. Restore the removed subnet entry, or introduce moved blocks that map the old addresses to the new ones, then re-plan. The success criterion at this step is 0 to destroy. Nothing else proceeds until that holds. Reverting config beats applying because the config change was accidental and carries no operational value.

3. Reconcile the manual changes, one at a time, code-side. For the security group rule: add the rule to the code, re-plan, expect a no-op for it. For the instance class: change the code to db.r6g.4xlarge so code matches reality; this is a real, intended resize and can be applied later in a window. For the S3 bucket: use terraform import (or an import block in Terraform 1.5+) so Terraform adopts the existing bucket rather than creating a duplicate. Do not console-revert the instance class - that recreates the incident, and the drift reappears the next time the same alert fires.

4. Apply in a change window with a manual RDS snapshot taken immediately beforehand, and only the remaining intended diffs in the plan.

The general rule: bring code into agreement with reality via config change or import; treat plan-and-apply as the last step, never the mechanism for undoing a manual fix.

2. Which changes to the code and state eliminate this class of destroy, and how do you prove a plan is safe before it runs?

Read the worked answer

Code changes.

Replace count with for_each keyed on a stable identifier - subnet name or CIDR - so removing an element never re-indexes its siblings. This is the direct fix for the destroy.
Use moved blocks to migrate existing state addresses without replacement when refactoring resource keys; this is the supported mechanism precisely for this situation.
Add lifecycle { prevent_destroy = true } to stateful resources: RDS, S3, KMS keys, anything holding data. ignore_changes only for attributes genuinely owned outside Terraform, and sparingly - it hides drift.
Pin provider and module versions exactly in the prod workspace and upgrade deliberately; a floor constraint like ~> 4.0 lets a terraform init silently change behaviour between runs.

Proving a plan is safe. Generate machine-readable output, terraform show -json tfplan, and evaluate policy in CI before any human reads it:

Fail if any resource with a data-bearing tag or type has action delete or replace in the prod workspace.
Fail on any destroy of more than N resources.
Fail if the plan's provider or module versions differ from the pinned lock file.

Additionally: terraform plan -detailed-exitcode in a scheduled per-account drift job (exit code 2 = drift), with output posted to a channel; apply only from CI using an artifact produced from the same commit, so the plan a human approved is the plan that executes. That last point matters more than any individual policy rule - most destroy incidents come from someone applying a plan nobody reviewed.

3. How do you keep incident response fast while stopping unmanaged console changes, and what evidence would you give an auditor?

Read the worked answer

The mistake is treating speed and control as opposites. Build a fast sanctioned path.

Break-glass access. Remove standing console admin in prod and replace it with a time-boxed, MFA-protected role (IAM Identity Center permission set with a 1-hour session) that any on-call engineer can assume by opening an incident in the ticketing system. The request, the approver, the session, and the CloudTrail events are all recorded automatically. Read-only access stays always-on so people can investigate without escalating.

Reconciliation SLA. Any console change made under break-glass must be reflected in Terraform within 24 hours by a PR linked to the incident. The drift job creates that ticket automatically from the diff, so it does not depend on the engineer remembering. Resources carry managed_by=terraform, owner, and cost_center tags; untagged resources are treated as findings.

Enforcement without heroics. If a resource has drifted for more than 24 hours, block the next deploy to that environment. That converts reconciliation from advisory to structural, without ever leaving an incident responder stuck.

Audit evidence to retain and hand over:

CloudTrail events correlated to the incident ticket ID, showing who assumed which role and when.
Terraform plan and apply logs, with the approver identity and the commit SHA, retained per release.
Git history for every reconciliation PR, linked to the incident.
Drift reports over time, showing detection-to-remediation time per finding.
Policy check results (which guardrails ran, what they found) attached to each release record.

That bundle answers the auditor's real questions - who changed it, was it authorised, was it reviewed, and how quickly was it corrected - without asking engineers to write a single extra document.