AWS DevOps Engineer Professional: Drift Between Terraform State and a Hotfixed Security Group

Drift Between Terraform State and a Hotfixed Security Group

A platform team manages 300 resources in Terraform with remote state in S3 and DynamoDB locking. During a Sev-1, an on-call engineer opens the console and adds an inbound rule to a production security group to restore access, then goes to bed. Two weeks later a routine terraform plan in CI shows the security group will be modified: Terraform wants to remove the manually added rule. The CI job runs terraform apply -auto-approve on merge to main. The engineer who made the hotfix has left the team. Nobody remembers the rule's purpose, but a grep of the incident channel shows it was for a partner IP that is still in active use. The team wants to prevent this class of problem without blocking emergency console changes.

1. The next merge to main would silently remove a rule that production depends on. What should happen before that apply runs, and how should the process be changed?

Read the worked answer

The plan is telling the truth: state and reality diverged and the next apply will reconcile toward code, deleting a live dependency. The apply must be blocked until the drift is understood and reconciled deliberately.

Process changes:

Make plan output a reviewable artifact. CI should post the plan and require human approval when the plan touches security groups, IAM, or networking in prod, rather than -auto-approve on every merge. A plan diff that removes a rule is exactly the case that needs eyes.
Reconcile the drift into code first. Add the partner rule to the Terraform config (or a variable), apply so state matches reality intentionally, then proceed. Never let an automated apply erase an unexplained production rule.
Emergency changes must be captured. The hotfix should have created a ticket/follow-up to codify the change within N days; the incident runbook should require it.

The core principle: -auto-approve is fine for low-blast-radius changes, but destructive plans on security-relevant resources need a gate.

2. Design a control that detects console drift quickly and forces emergency changes back into code, while still allowing break-glass access.

Read the worked answer

Layer detection and reconciliation:

Scheduled drift detection: run terraform plan -detailed-exitcode on a schedule (e.g., hourly) against prod; exit code 2 means drift. Publish the result as a metric and alarm on it. This catches the console change within an hour instead of two weeks.
Break-glass with attribution: allow console changes only through a documented break-glass role that requires an incident ID, is MFA-gated, and emits an event to a log/queue that opens a follow-up task automatically. The change is permitted but never invisible.
Config rules as a backstop: AWS Config rules (e.g., detecting security group changes) plus CloudTrail alerting on AuthorizeSecurityGroupIngress outside the pipeline role give near-real-time signal independent of Terraform.
Reconciliation SLA: any detected drift must be either codified (import/update config) or reverted within a defined window; the drift alarm stays red until resolved.

This keeps emergency access fast while making drift a loud, owned event rather than a latent surprise at the next apply.