Stale Outputs, Duplicated Credentials, and an 11-Minute Queue

Why it happens: terraform_remote_state is a data source evaluated at plan time against whatever state snapshot exists then. Nothing in the config knows that net-prod is running. app-prod also triggers on VCS pushes, so a net-prod apply can be executing while app-prod plans - the classic empty-VPC-ID symptom, since the output either does not exist yet or is the pre-apply value. An apply approved from a stale plan makes it worse: the plan can be hours old by the time it is confirmed.

The options:

1. Run triggers - app-prod queues only after net-prod completes successfully. Correct ordering, but it does not stop a concurrent VCS-triggered run; you must also disable VCS-driven auto-apply on app-prod or restrict it to the trigger path, otherwise the two paths race.
2. tfe_outputs data source - reads the producing workspace's current state via the API at plan time. It removes the hand-rolled remote_state config and gets you org-level consistency, but it is still a plan-time read, so it does not fix ordering on its own.
3. A single workspace for both - removes the boundary but reinstates the blast radius you split them to avoid.

Recommendation: run trigger plus tfe_outputs, with app-prod's VCS auto-apply disabled so the trigger is the only entry point. Then confirm the empty-ID class of failure disappears in run history before declaring it fixed - the failure was intermittent, so one green run proves nothing.

Precedence: workspace-level variables override variables from variable sets, and more specific scope (project) overrides organization-wide. Two consequences: when consolidating 14 copies into a variable set, a leftover workspace-level override silently wins and the rotation does not propagate - audit for duplicates before rotating. And a variable defined in multiple variable sets is a precedence puzzle nobody should have to solve at 03:00; keep exactly one definition per name.

Exposure: marking a variable sensitive hides it in the UI and run output from that point forward, but the values were already in plain text and in run logs. Treat them as compromised: rotate, not just re-flag. Anyone with read access to those two workspaces has seen them.

Long-term: stop shipping static keys at all. Use HCP Terraform's dynamic provider credentials (AWS OIDC trust to a role scoped per project), which removes the secret from the platform entirely and reduces 14 rotation targets to 0.

Caveat worth stating: marking a variable sensitive does not keep the value out of state. If the credential is written into a resource argument, it is in the state file regardless. Sensitivity is an output filter, not state encryption.

Blocking requires policy as code (Sentinel or OPA policy sets), evaluated after plan and before apply. Set the policy to hard-mandatory so a violation cannot be overridden by anyone; soft-mandatory allows an org owner to override, which is usually what you actually want for a new policy - start advisory, watch false positives for two weeks, then promote. Important behaviour to plan for: a policy failure still consumed a plan, and the run shows as policy-overridden/failed rather than a normal apply.

Slack belongs in a run task, which calls out to a webhook at pre-plan, post-plan, or pre-apply. Run tasks are either advisory (log only) or mandatory (a non-success blocks the run). For a notification you want advisory, attached at post-plan on the project scope for prod workspaces only.

Cost, given 3 runners and an 11-minute mean queue: every run task is an extra remote round trip inside the run, and mandatory ones serialize the run further. Attaching a Slack task to all 14 workspaces adds latency to every run for information nobody reads on non-prod. Scope it to the production project. If the queue is already the bottleneck, the fix is more concurrency (more runners or larger agents) before adding more per-run work - otherwise you will have excellent notifications about runs that start 15 minutes late.

Finally: the policy needs a policy set attached at org or project scope; a policy that exists but is not attached to the workspace enforces nothing.