Provider Upgrade Turns a Routine Plan Into 212 Changes
A B2B SaaS company runs 38 HCP Terraform workspaces against AWS. Each environment has its own workspace, remote state, and a GitHub-driven VCS connection. The platform team pins Terraform at 1.5.7 and the AWS provider at ~> 5.0. Last week an engineer upgraded the provider to 5.62 in a staging workspace, ran terraform apply from a laptop after the remote run stalled on an orphaned state lock, and then opened a pull request. The next plan in the shared prod-network workspace shows 212 changes: 180 tags_all updates, 18 security group rule diffs, and 6 -/+ replacements on route table associations. No application code changed. The orphaned lock is still visible, and two other workspaces show the same provider version. The team wants to merge before the Friday release. The release includes a new customer-facing endpoint and cannot slip.
1. What is the most likely explanation for the 212-change plan, and what should the team do before approving anything?
Read the worked answer
The provider upgrade is the trigger, not the orphaned lock. A newer AWS provider can change default tag handling and computed attributes, so existing resources show tags_all diffs even when configuration is unchanged. The laptop apply also means state may have been written outside HCP Terraform policy and audit controls. The six replacements need separate scrutiny because route table associations can cause traffic loss.
Before approving, the team should: (1) confirm no active run owns the lock, then release it through the UI with an audit note; (2) run a refresh-only plan in each affected workspace to separate real drift from provider-schema noise; (3) compare the provider version and lock file across workspaces; (4) inspect the six replacements with terraform show -json and decide whether they are schema-driven or caused by manual changes; and (5) test the upgrade in a non-production workspace with the same module version. Applying the 212-change plan directly would mix an infrastructure upgrade with a release and make rollback ambiguous.
2. How should the team safely reconcile the six replacements and the 18 security group diffs?
Read the worked answer
Treat the replacements and security group diffs as two different problems. For the route table associations, determine whether the provider now represents the same relationship with a different resource address or whether the configuration genuinely changed. If it is an address change, use moved blocks or terraform state mv in a reviewed change rather than allowing destroy/create. If replacement is unavoidable, use create_before_destroy only where the resource supports it and schedule a maintenance window, because route replacement can black-hole traffic.
For the security group diffs, compare the plan against the last approved commit and against AWS Config history. Manual console changes should be imported into Terraform or deliberately reverted, not blindly overwritten. Apply the provider upgrade first in staging with the same module and variable values, then promote the lock file. Keep the customer-facing endpoint release in a separate pull request so a failed provider migration does not block or entangle the release.
3. What guardrails would prevent a repeat of the laptop apply and provider drift?
Read the worked answer
The core guardrail is to make HCP Terraform the only execution path for shared and production workspaces. Remove standing admin tokens from laptops, require SSO team membership, and restrict who can unlock state. Set workspace execution mode to remote; local CLI applies should be limited to sandbox workspaces with no production credentials.
Add policy checks that fail a run when the provider version differs from the approved lock file, when a plan contains destroy/replace actions in production, or when the run was not triggered from VCS. A run task can inspect the plan JSON for replacements and require approval. Use separate workspaces or stacks for network foundations so a provider bump cannot combine with application changes. Finally, configure lock timeout alerts and a documented lock-recovery procedure so engineers do not bypass remote runs when a run stalls.