Splitting Prod State Out of a Shared Workspace After a Bad Unlock

Order matters more than the commands.

1. Freeze the source first. Remove write permission on the old bucket's env:/prod/ prefix from the CI role and every human role. A migration that can be raced is not a migration.
2. Extract the state. terraform workspace select prod (in the frozen window) then terraform state pull > prod.tfstate. Each workspace is its own state object, so this file contains only the 190 prod resources - no filtering needed. Keep the file out of Git.
3. Reconcile the partial apply before moving it. The interrupted run left resources created but not recorded. Read the audit trail (CloudTrail/apply logs), then terraform import or remove the orphans, and confirm the frozen window's plan is clean. Migrating a corrupt state just relocates the corruption.
4. Create the new root. Because the audit wants separate credentials, use a separate directory with its own backend "s3" block (new bucket, new key, new role), not another workspace in the same config. Addresses only survive if the module path and resource names are identical - if the new directory calls a module where the old root had resources at the root, you need moved blocks, and the plan will tell you.
5. Push and verify. terraform init against the empty new backend, terraform state push prod.tfstate, then terraform plan - it must be empty. Only then delete the old workspace and, later, the old state object.

Rollback is the backup file plus the still-intact old bucket object; keep both until the first successful apply under the new role.

force-unlock releases the lock record. It does not cancel a running apply and it does not repair state - it only removes the signal that says "someone is working." On-call used it based on a stale-looking pipeline, while a runner was still between API calls. The result was two writers on one state object, which is the one failure mode S3+DynamoDB locking exists to prevent.

The decision rule: unlock only after positively confirming no process holds the lock - check the DynamoDB item's Info (ID, who, when), the CI run's actual status in the pipeline UI, and whether the runner is still alive. If any of those is unverified, wait or kill the runner first. A lock that expires on its own is a signal, not an inconvenience.

Prevention, in priority order:

• A CI concurrency group (one apply per state object) so a retried job cannot overlap a still-running one.
• -lock-timeout=5m on plan/apply so transient contention waits instead of failing fast and tempting a human to unlock.
• Give the DynamoDB table a CloudTrail alarm on DeleteItem so every unlock is visible.
• Kill zombie runners automatically; a dead runner holding a lock is the only legitimate unlock case, and it should be a documented runbook step, not an on-call improvisation.

Workspaces are a naming convention, not a boundary - terraform.workspace is a string in the resource address, and a typo selects a different state with the same config. Structural fixes, cheapest first:

1. Directory per environment (envs/dev, envs/prod) with distinct backend blocks. Prod then requires a deliberate cd into a directory whose backend points at a bucket the dev role cannot write. This is what the audit migration is already moving toward; finish it.
2. Trust-policy separation. The prod S3 prefix and the prod provider role are assumable only by the production pipeline identity, so a dev shell with a typo gets AccessDenied rather than a plan against prod state.
3. A runtime assertion for the transition period - a check/precondition comparing terraform.workspace (or a var.environment) against a value baked into the backend config, so a mismatch fails the plan with a clear message instead of touching resources. Treat it as scaffolding to remove once directories land, and remember it fails the plan only when it is a precondition, not a check.

What I would not rely on: -target, variable defaults per workspace, or reviewer discipline. All three are conventions that fail at 02:00.