I85PREP
⌘ K
Terraform Associate study materials
Terraform Associate study resources

Two Applies, One State File

Terraform Associate · Put your knowledge to work in context.

All case studies

Two Applies, One State File

A six-person platform team runs about 140 AWS resources from one root module with an S3 backend and DynamoDB state locking. CI applies on every merge to main, but engineers also apply from laptops for hotfixes. Last Thursday an engineer ran apply from a laptop while a CI run was mid-apply. Now plan on main proposes creating 11 security groups and 4 S3 buckets that already exist in the account, and an apply of that plan fails with AlreadyExists. DynamoDB shows no lock entries today, the S3 state object was last modified from the laptop, and nobody has ever run terraform state list against production. The state bucket has versioning enabled and 213 retained object versions.

1. Explain what most likely happened to the state object, and list the first four checks you run before changing anything.

Read the worked answer

Diagnosis. Bucket locking serialises writes; it does not merge them. Each apply read the same serial and then wrote a whole new state object. The last writer (the laptop) won, and any resource created by the CI run that the laptop's copy had not seen is now missing from state. Those 11 groups and 4 buckets are real but untracked, which is exactly why plan wants to create them and apply dies on AlreadyExists.

A second candidate must be ruled out: the laptop may have run without the backend configured (a stray local terraform.tfstate), in which case the two runs never shared state at all.

First checks, in order:

  1. terraform state list | wc -l on main, compared against the ~140 resources the team believes it has.
  2. aws s3api list-object-versions --bucket <bucket> --prefix <key> to get VersionIds and timestamps around the incident.
  3. terraform state pull > current.json on a scratch copy; inspect serial and lineage. A different lineage means a genuinely different state file, not merely an older serial.
  4. Pull the two or three candidate versions and diff their resource address lists to see exactly what each run was tracking.

Do not run apply to make the error go away - it creates shadow resources and buries the evidence.

2. Write the reconciliation plan, from frozen writes to a clean plan, and the cutover that stops this recurring.

Read the worked answer
  1. Freeze writes. Remove s3:PutObject on the state key from every human role; only the CI role keeps it. Keep versioning and add MFA delete.
  2. Pick the base version. From the version listing, choose the object version whose resource list is the best superset. Restore it to a scratch key, never in place, and plan against it with -state or by pointing a throwaway backend at the copy.
  3. Classify every orphan. For each resource the plan wants to create, confirm the real object's identity (tags, ARN, creation date). Two buckets may be duplicates from the two racing runs; decide which survives before importing anything.
  4. Import under proper addresses. Use import blocks so the action is reviewable in a PR (import { to = aws_s3_bucket.dump, id = "..." }, and repeatable form with for_each for the security groups). CLI terraform import works but leaves no audit trail.
  5. Import, then plan to no-op. An import only writes state; the config must already match reality or the next plan proposes changes. Iterate until plan reports no changes, then delete the temp state copies.
  6. Cutover. Make the CI role the only writer, add a CI concurrency group so two pipelines cannot overlap on the same workspace, replace any -lock=false with -lock-timeout=5m, and alarm on state object writes that do not come from CI.

3. CI's apply had been running for 40 seconds when the laptop apply started, yet no lock error appeared. How is that possible, and which explanations point to a bigger problem than a race?

Read the worked answer

Locking is only enforced by the backend the run actually uses. Plausible explanations, roughly in order of likelihood:

  • Different lock key. DynamoDB locks are keyed by bucket plus state key. A laptop running terraform workspace select staging or a different key in the backend block locks a different item and never contends with CI's production apply.
  • Different backend entirely. No backend block (or an overridden -backend-config) means local state; the lock table is never contacted.
  • -lock=false on the laptop. Someone was in a hurry. This should be treated as a policy violation, not a mistake.
  • Missing permissions. A role without dynamodb:PutItem on the lock table fails loudly, so this is unlikely to be silent.

The first two are the serious ones: they mean the lock never had a chance to help, and the same setup could silently produce divergent states every day. Audit the backend block across every directory and check CI logs for Acquiring state lock before concluding it was a timing accident. Fix the process, not just the state file.