Terraform State Collision Across ACI Environments
A platform team manages 30 ACI tenants with Terraform. Prod and non-prod use the same S3 bucket and the same Terraform workspace; the S3 backend has no DynamoDB table and no use_lockfile. Two GitLab pipelines, one for each environment, can run at the same time. The repo does not commit .terraform.lock.hcl, and required_providers allows ~> 2.0 for the ACI provider. Last Tuesday, a non-prod pipeline refreshed state while a prod pipeline was applying an EPG change; the prod apply failed with a stale state error, then a retry reported 0 to add, 0 to change, 0 to destroy even though the EPG was absent. A later provider upgrade changed an attribute default and produced a plan that would have replaced a production contract. The team needs to make changes auditable and prevent concurrent state mutation without rebuilding every tenant.
1. Explain the failure chain that produced the stale state error and the later 0/0/0 plan. What immediate recovery steps would you take before allowing another apply?
Read the worked answer
The failure is shared mutable state plus no locking. Both environments point at the same state key/workspace, so the non-prod refresh and prod apply race. Terraform state serial increases on write; the prod apply used an older serial and failed. The retry read a state written by non-prod that already contained the EPG address, so plan reported no changes even though APIC did not have the object. The provider upgrade then introduced a new default, causing a replacement plan on a contract. Immediate recovery: freeze both pipelines; enable S3 versioning if not already on; restore the last known-good state version before the race; run terraform state list and compare every prod resource with APIC (or use terraform plan -refresh-only and inspect). Do not run apply until state and APIC agree. If the EPG is missing in APIC but present in state, remove it from state and import the real object or recreate it deliberately. If an object exists in APIC but not state, import it. Record the corrected state version and require a reviewed plan.
2. Design the backend, workspace, provider, and CI isolation model that prevents this class of collision. Be specific about state keys, locking, and provider version control.
Read the worked answer
Backend isolation: give each environment its own state key, for example env/prod/terraform.tfstate and env/nonprod/terraform.tfstate, and preferably separate AWS accounts and buckets. Enable S3 versioning, SSE, and access logging. Add state locking with a DynamoDB table (dynamodb_table) or, on Terraform versions that support it, S3 native locking (use_lockfile=true); do not rely on CI concurrency alone. Add a CI concurrency group per environment, but treat locking as the source of truth. Stop using one workspace for both environments. Pin the ACI provider to an exact version in required_providers, commit .terraform.lock.hcl, and use terraform providers lock for the platforms CI uses. Provider upgrades happen in a dedicated PR and are tested in non-prod first. Use separate APIC credentials and variables per environment; never put credentials in state. This prevents a non-prod run from refreshing or writing prod state and makes provider behavior reproducible.
3. How should the CI pipeline validate and apply plans so a provider default change cannot silently replace a production contract?
Read the worked answer
CI validation: run terraform fmt -check, terraform validate, then terraform plan -out=tfplan and upload the plan artifact. Never let a later job re-plan; apply the saved plan with terraform apply tfplan after manual approval. Parse terraform show -json tfplan and fail the job if any prod ACI contract or EPG has an action containing delete or replace, or if the plan touches resources outside the approved address list. Post the plan summary to the MR. For provider upgrades, run the same plan against non-prod and compare resource actions; require a second reviewer for any replacement. Add a drift check on a schedule using terraform plan -refresh-only -detailed-exitcode; a non-zero exit opens a ticket instead of auto-applying. For state repair, use terraform import and terraform state rm only through a reviewed runbook. This makes the change auditable and prevents the earlier silent 0/0/0 apply.