One Terraform State File for 1,200 Resources
The platform team manages 1,200 Terraform resources in a single S3 state with DynamoDB locking. Six engineers across three squads apply from laptops using long-lived admin access keys. A plan takes 22-28 minutes because every provider and data source is refreshed. Last month an engineer renamed a security-group module; the plan showed one destroy and one create, and the apply replaced a security group attached to the production RDS instance, causing 40 minutes of connection failures. Nobody read past the summary line. The same state holds VPC, IAM, RDS, EKS, and DNS for three environments behind workspaces. Console hotfixes during an incident produced drift that reappeared as a surprise revert two weeks later. There are no policy checks and no required approvals; the only gate is a Slack message. Leadership wants faster plans and no repeat of the security-group outage without adding a week of process.
1. Explain why a single state file causes both slow plans and excessive blast radius, then propose a state segmentation that addresses both. What new problems does your segmentation create?
Read the worked answer
Why one state hurts on both axes. Terraform refresh is proportional to the number of resources in the state, so 1,200 resources across five providers means every plan re-reads everything, even for a DNS change. That is the 22-28 minutes. Locking makes it worse: one lock serializes six engineers, so a stuck apply blocks every change. Blast radius comes from the same root cause: one plan graph spans VPC, IAM, RDS, EKS, and DNS, so a rename in a module can appear as a destroy adjacent to production data. Nothing about the tooling forces the reviewer to notice, and with a 25-minute plan people stop reading.
A workable split. Segment first by environment and account, because those must never share a failure domain. Within an environment, split by lifecycle and change frequency:
- network: VPC, subnets, route tables, TGW attachments (changes monthly)
- data: RDS, S3, KMS, backups (protected, prevent_destroy)
- platform: EKS, IAM roles, node groups (changes weekly)
- edge and app config: DNS, ALBs, SSM parameters (changes daily)
Cross-stack references go through remote state outputs or data sources, never direct resource references. Ordering is enforced in CI: network plans and applies before platform, platform before app.
Costs of the split. More pipelines and more state backends to secure. Cross-state references are a hidden coupling: an output rename breaks consumers, and a producer apply can invalidate a consumer plan mid-flight. Dependency ordering must be explicit and tested. Outputs need versioning and a deprecation path. Also expect a one-time migration risk, discussed separately. Five to eight states per environment is usually the sweet spot; going to one state per resource just moves the pain into orchestration.
2. Design the guardrails that would have prevented the security-group replacement, using CI-enforced controls rather than reviewer diligence.
Read the worked answer
Make the plan the artifact, not the summary. CI runs terraform plan -out=tfplan, uploads the binary plan, renders a full resource-change list in the PR, and applies only that exact saved plan after approval. Nobody applies from a laptop; laptops have read-only or no credentials. Authentication uses OIDC federation to short-lived role credentials, so the long-lived admin keys disappear.
Policy as code at plan time. Run OPA/Conftest or Sentinel against the JSON plan and hard-fail on: any destroy of a resource type in a protected list (security groups in use, RDS, S3 buckets with data, KMS keys, stateful sets); public S3 ACLs; security groups open to 0.0.0.0/0 on non-443 ports; missing required tags; IAM policies with wildcards. A deny is not overridable by a reviewer; an exception requires a documented, time-boxed waiver in the policy repo.
Make renames non-destructive by design. Require moved blocks for any address change and import for adopting existing resources, so a module rename becomes a state move rather than destroy/create. Add lifecycle { prevent_destroy = true } to data-bearing resources as a second, independent net.
Approval gates proportional to risk. A plan with zero destroys and under a size threshold auto-applies after green tests. A plan containing a protected-type destroy or an IAM change requires a named approver from the owning squad plus a second reviewer. This keeps small changes fast while putting friction exactly where the outage came from.
Drift control. Nightly -refresh-only plans per stack open a ticket when drift appears, and console write access is removed from human roles so hotfixes go through the pipeline. That closes the loop that produced the surprise revert.
3. You must split the state with zero downtime. Give the migration sequence, the safety checks at each step, and the point at which you abort.
Read the worked answer
Prepare. Freeze applies for the window (announce it, use the lock). Snapshot the S3 state object and enable versioning; confirm DynamoDB locking works for the new backends. Build the target stack skeletons with the same backend pattern but distinct keys, and no resources yet. Map every resource address with terraform state list and draw the dependency edges you need to preserve as outputs.
Split per stack, smallest first. For each resource: terraform state mv from source to target, or in newer Terraform use a removed block in the source and an import block in the target so the move is reviewable in code. Remove from the source before applying the target so no window exists where two states manage the same object. Never run source and target applies concurrently.
Verify after each move. In the target, terraform plan must show no changes for the moved resources; any create/destroy means the address or config does not match and you stop. In the source, plan must show only the removals you intended. Then destroy nothing.
Establish outputs before consumers. Create the remote state outputs in the producer stack and switch consumers to read them, one consumer per PR, while the old direct references still work. Only after all consumers read outputs do you remove the old resources from the monolith state.
Ordering. Do dev, then staging, then production, with at least a week between production and the previous environment so latent issues surface. Keep the source state read-only but intact for 30 days.
Abort criteria: any unexpected create or destroy in either plan, any moved resource that reappears as tainted, a lock contention that exceeds the window, or a consumer plan that fails to resolve outputs. Abort means restoring the state version from S3 and re-running plans to confirm zero diff; that is safe because no resources were destroyed at any point.