Console Drift During a Payment Incident
A platform team manages three AWS accounts (dev, staging, prod) with Terraform. Production has 240 resources, remote state in S3 with DynamoDB locking, and changes go through a pipeline that runs plan on pull request and apply after approval. During a payment incident, an engineer changed a production security group and an RDS parameter group in the console to restore traffic. The next plan shows the security group will be reverted and the RDS parameter group will be replaced because its engine version was also changed manually. The team has a 30-minute maintenance window and cannot tolerate a database replacement. Two engineers propose: (a) run terraform refresh and apply, (b) delete the resources from state and import them back after the incident. The Terraform version is 1.7, providers are pinned, and only three people have prod console access.
1. Assess the two proposals and describe a safer immediate path to reconcile state without causing an RDS replacement.
Read the worked answer
Proposal (a) is incomplete. terraform refresh, or the modern refresh-only apply, updates state to match reality; it does not change configuration. A normal apply would then try to restore the declared security group rules and may replace the RDS parameter group because the engine version is an immutable or replacement-triggering argument. Proposal (b) is unsafe: deleting a managed resource from state does not change the configuration, removes Terraform's protection, and can orphan or destroy data if the import is imperfect. Import is for bringing existing resources under management, not for avoiding a replacement plan.
The safer immediate path is to freeze applies, take an RDS snapshot, and export the current console settings. Open an incident PR that codifies the security group change and the parameter group values. Run a targeted plan for each resource and inspect terraform state show plus the provider schema to identify the argument forcing replacement. If the forced replacement is the engine version, do not apply. Restore the engine version in the console to the previously managed value, or move the upgrade to a planned blue/green or maintenance procedure. Add prevent_destroy and RDS deletion protection before any further apply. Use ignore_changes only as a short-lived, ticketed measure with an expiry, never as permanent drift tolerance. Then apply the targeted change during the maintenance window and verify with a post-apply plan.
2. What controls would prevent console drift and make future emergency changes auditable?
Read the worked answer
The controls should make the console path deliberate and observable rather than routine. Use IAM Identity Center permission sets instead of long-lived console users, and apply SCPs that deny production write actions except to a break-glass role. The break-glass role should require MFA, a ticket or incident ID, and session logging; CloudTrail and AWS Config should alert on its use and on changes to security groups, RDS parameter groups, and IAM. Enable drift detection and run a scheduled plan that reports differences before the next release.
For normal changes, require pull requests and a pipeline that runs plan, policy checks, and apply. Use state locking, pinned providers, and environment-specific approvals. Mark stateful resources with prevent_destroy and deletion protection, and test restores. For emergency changes, require a post-incident codification deadline, such as 24 hours, and track it as an action item. The engineer who made the console change should not be the only approver of the codification PR.
The audit evidence should include CloudTrail events, the incident ticket, the break-glass approval, the Terraform plan and apply logs, and the final post-apply plan showing no unexpected drift. That closes the loop between the emergency action and the declared infrastructure.