Terraform Drift and a Laptop-Held Production State
Orbit CRM runs three Terraform environments. Dev and staging use an S3 remote backend with DynamoDB locking; production state is stored only on one engineer's laptop because the team was afraid of a bad migration. Terraform 1.6 manages 74 production resources, including an RDS instance, an ALB, and security groups. During a Sev-1 incident, an engineer opened port 3306 to 0.0.0.0/0 directly in the AWS console and added a new listener rule manually. The next terraform plan from the laptop shows 12 changes, including replacement of the RDS instance because a subnet or parameter drifted. A separate engineer has edited the same module in Git, but not applied it. A customer-facing release is due in two days. The production account has no SCPs restricting console writes.
1. What should the team do before running terraform apply, given the production state is on a laptop and the plan shows RDS replacement?
Read the worked answer
Do not apply that plan. First back up the laptop state file and .terraform directory to encrypted storage, then migrate state to an S3 backend with versioning and DynamoDB locking using terraform init -migrate-state. Confirm the state serial and resource count before and after migration. Run terraform plan -refresh-only to separate real drift from desired configuration changes. For the manual listener rule, use terraform import or remove it in code; for the open security group, revert it through code. Investigate the RDS replacement cause with terraform state show and the AWS console before approving anything. A laptop state plus a replacement plan for a production database is a data-loss risk, so the immediate goal is a locked, backed-up, reviewed state, not speed.
2. How should the two-day customer-facing release be handled?
Read the worked answer
Freeze infrastructure changes and ship only the application release if it does not depend on the drifted infrastructure. Reconcile the security group through code and import the listener rule so the next plan is clean. Defer any RDS replacement to a maintenance window with a tested restore plan; replacing a database because of a subnet or parameter drift is not a release task. If the release requires infrastructure changes, run the plan from CI with a reviewer, not from the laptop, and attach the plan output to the change record. Prepare a rollback for the application and confirm database backups. The release decision should be based on whether the infrastructure change is understood, not on the calendar alone.
3. Design the state and pipeline model that prevents this recurrence.
Read the worked answer
Use one remote backend per environment and account, with S3 versioning, SSE-KMS encryption, DynamoDB locking, and access limited to the CI role and a break-glass role. Run terraform plan on pull requests and apply only from CI after approval, with no console writes in production. Add scheduled drift detection that opens a ticket when plan -refresh-only changes. Pin Terraform and provider versions, pin module versions, and store modules in a reviewed repository. Use separate workspaces or directories for dev, staging, and production, not one state file. Enforce SCPs or IAM policies that deny console changes to production infrastructure except for a logged break-glass role. Document an emergency change process that requires the same code update and import within a defined window.