Out-of-Band Firewall Rule, Unlocked State, and Two Concurrent Applies

Terraform's drift detection is only as wide as the provider's schema. The resource representing the policy object declares a set of attributes; on refresh, the provider calls the controller's read API and populates those attributes. If the extra rule exists as a child element the provider does not model - or is attached in a way the read does not return - then refresh sees exactly what it expects and the plan is clean.

The general property: state is a cache of the last observation the provider was capable of making, filtered through a schema someone wrote. "No diff" therefore means "no difference in the modelled surface," not "the device matches the repository." Three consequences follow, and they are what the exam is really testing:

• Coverage gaps are invisible by construction. You cannot detect drift in something nothing claims to own. This is why the estate needs a separate reconciliation against a full export of the controller's actual configuration, not just terraform plan.
• terraform plan is not an audit. It answers "will Terraform change anything?" It does not answer "does reality match intent?"
• Import is the tool for closing the gap. Adopting the unmanaged element into state makes it modelled and thereafter drift-visible; leaving it unmodelled guarantees the same surprise again.

So the Wednesday plan was correct and useless. Real assurance requires comparing the controller's exported configuration to intent, independent of what the provider chose to read.

The ordering rule is: stop making it worse, establish ground truth, then fix the mechanism. Restoring control beats restoring tidiness.

Today - stop the bleeding.

1. Disable local applies: restrict production controller credentials so only the CI identity can write. Remove engineer-held production write tokens from laptops.
2. Freeze the pipeline until state can be trusted; a scheduled apply against an unknown state can compound the problem.
3. Snapshot the S3 state object (versioning plus an out-of-band copy) so nothing can overwrite the only record.
4. Export the controller's live firewall configuration as the authoritative "what is enforcing right now" artifact.

This week - establish ground truth and fix the mechanism. 5. Diff the export against the last known-good intent. Classify every discrepancy: legitimate emergency rule to keep (import it), leftover to remove (delete via change control), or unexplainable (investigate). Do not skip the unexplained bucket. 6. Add a DynamoDB lock table to the S3 backend - this is a one-line fix and its absence allowed the interleaved applies. 7. Separate state per data center and per environment so a production apply cannot touch three sites at once. 8. Make CI the only apply path, with a plan artifact reviewed in the pull request.

This month - prevent recurrence. 9. Controller-level RBAC: GUI write in production removed; break-glass account with alerting and an automatic ticket. 10. Scheduled drift detection that compares a full export to intent, reporting unmanaged elements, not just provider-modelled ones. 11. Require short-lived credentials and reject applies from branches older than the current main.

The lock fixes one of the three contributing causes, and the least interesting one.

What the lock does fix. A DynamoDB (or equivalent) lock makes the second apply fail fast instead of interleaving two writes against the same state. That is real and necessary - concurrent applies can corrupt state and the consequences are silent. Keep it.

What it does not fix.

• Stale intent. The engineer's branch was six days old. A lock serialises the two applies; it does not make the second one correct. The second apply would have run just as successfully a few hours later, and still recreated 40 address objects from outdated definitions. The defect is applying a plan built from old inputs, not applying it concurrently.
• Who is allowed to apply. The engineer applied from a laptop, outside CI, with credentials that permitted it. That path exists with or without a lock and is the reason the incident was possible at all.
• Blast radius. One apply touching three data centers' worth of objects is a design choice; the lock does not reduce it.

What actually closes the class of incident: CI-only applies with short-lived credentials, branch protection so stale branches cannot reach an apply, plan-and-apply of the same saved plan file so what was reviewed is what executes, and per-site state so a mistake lands in one place. The lock is one of five controls, and it is the one that a checklist can satisfy without changing any behaviour - which is exactly why it should not be accepted as the whole answer.