Static Cloud Keys and Auto-Apply Undermine an HCP Terraform Pipeline
A platform team runs 22 HCP Terraform workspaces for AWS infrastructure. GitHub Actions triggers runs through the HCP Terraform API. AWS access keys are stored as GitHub organization secrets and passed into workspaces as environment variables. The pipeline processes about 85 runs per day, has a workspace concurrency limit of 10, and takes 18 minutes end to end. A policy run task checks plans, but production workspaces have auto-apply enabled. Last week a plan that failed the policy task was still applied because an engineer re-queued the run after a transient API error. The security team wants to remove static keys, and engineering wants fewer queued runs without weakening production approval.
1. How should the team replace the static AWS keys with short-lived credentials in HCP Terraform?
Read the worked answer
Use HCP Terraform dynamic provider credentials with AWS OIDC. Configure AWS to trust the HCP Terraform OIDC provider, create an IAM role per workspace or per environment, and set the workspace environment variables that tell Terraform to assume that role during runs. Scope each role's trust policy to the specific HCP Terraform organization, workspace, and run phase so a plan in one workspace cannot assume a production apply role. Remove the GitHub organization secrets and stop passing AWS keys into workspaces.
Validate by running a plan that calls aws sts get-caller-identity and confirming the assumed role ARN. Check that the role cannot be assumed outside HCP Terraform and that plan and apply use separate permissions where possible. Keep break-glass credentials outside the pipeline with strong audit. This removes long-lived keys from GitHub and reduces blast radius if a repository or runner is compromised.
2. Why are runs queuing, and what changes would reduce queue time without allowing unapproved production applies?
Read the worked answer
Eighty-five runs per day is not high in absolute terms, but the 18-minute duration and concurrency limit of 10 create bursts. Queueing is likely caused by multiple workspaces competing for the same concurrency pool and by runs being triggered on every commit rather than on relevant path changes. Check the run duration breakdown: queue time, plan time, policy task time, and apply time. If policy tasks are serialized, that can dominate.
Reduce queue time by using run triggers so only dependent workspaces run after their upstream succeeds, by filtering VCS triggers to relevant directories, and by separating plan and apply workspaces where appropriate. Increase concurrency only after checking the cloud API quotas and state-lock contention. Do not enable auto-apply in production to shorten the pipeline. Instead, keep production applies behind manual confirmation or a deployment gate, and reserve concurrency for production workspaces during release windows.
3. What governance change prevents a failed policy run from being re-queued into an apply?
Read the worked answer
The immediate flaw is that auto-apply can proceed after a policy failure when a run is re-queued. Production workspaces should not use auto-apply. Require a successful policy task, a successful plan, and an explicit approval by an authorized team before apply. Configure the run task as advisory only where appropriate; for production it must be mandatory. Re-queuing a run should create a new plan and re-run policy checks, not resume an old apply.
Add Sentinel or OPA policy that blocks applies when the policy evaluation is missing, stale, or failed. Use HCP Terraform teams and permissions so the engineer who re-queues cannot also approve without a second identity. Enable notifications and audit log streaming for policy failures and apply events. Test the control by deliberately failing a policy in a non-production workspace and confirming that no apply path exists.