ECS Deployments: Overlapping Runs, OIDC Trust, and Environment Approvals
You deploy a containerized API to AWS ECS from GitHub Actions. deploy.yml triggers on push to main, assumes an IAM role via OIDC (role-to-assume: arn:aws:iam::123456789012:role/gha-deploy), builds and pushes to ECR, then runs aws ecs update-service --force-new-deployment. Production uses a GitHub Environment named production with required reviewers. Last Tuesday, two PRs merged 40 seconds apart. Both deploy runs started; the second updated the ECS service while the first was still running a 12-minute smoke test. The smoke test passed for the first run after the second deployment, so the pipeline reported success for an older image. On Friday, OIDC failed for a pull_request workflow with 'Not authorized to perform sts:AssumeRoleWithWebIdentity'. The IAM trust policy currently has token.actions.githubusercontent.com:sub equal to repo:acme/api:ref:refs/heads/main and no conditions on aud.
1. Explain the overlapping deployment and how to prevent older runs from reporting success over newer deployments. Include concurrency configuration and image traceability.
Read the worked answer
Both merges created separate workflow runs. Each run built an image tagged with its own SHA, deployed it with force-new-deployment, and then ran a smoke test that queried the ECS service by name. The service is a shared mutable resource, so run A's smoke test hit run B's tasks and passed. The pipeline cannot infer which image it validated. Serialize production deploys with concurrency: group: deploy-production, cancel-in-progress: false so run B waits for run A. That prevents overlap but still lets an older SHA deploy, so add a guard: before deploy, compare github.sha with the current tip of main; if it is stale, exit successfully or fail as superseded. Capture the image digest from the build job as an output, deploy that digest, and smoke test a specific ECS task set or revision, not the service name. With cancel-in-progress: true, a newer run cancels the older one, but cancellation can interrupt a deploy; use it only if the deploy step is idempotent and cleanup is safe. Record the digest in the deployment environment URL and run summary.
2. The OIDC failure occurs on pull_request. Diagnose the trust policy mismatch and propose a least-privilege trust policy for main-branch deploys and PR plan-only jobs.
Read the worked answer
The PR job presents a different sub claim. For a pull_request event from the same repository, the sub is typically repo:acme/api:pull_request; for a fork PR it may include the fork owner/repo. The trust policy only allows repo:acme/api:ref:refs/heads/main, so STS rejects it. The missing aud condition is also a security gap. For main deploys, require aud equals sts.amazonaws.com and sub equals repo:acme/api:ref:refs/heads/main. If the job uses a GitHub Environment, the sub may be repo:acme/api:environment:production, so choose one and test. Do not use StringLike with repo:acme/api:*. For PR plan-only checks, create a separate role with read-only permissions and trust repo:acme/api:pull_request; do not attach deploy permissions. Both workflows need permissions: id-token: write, contents: read. Rotate any long-lived keys and remove them from secrets.
3. Required reviewers on production did not stop the race. Explain when environment protection rules apply and how to combine them with concurrency and branch protection.
Read the worked answer
Environment protection rules are evaluated when a job that references environment: production starts. Required reviewers must approve that job before it runs, but approvals are per job run; two runs can both be approved, and the environment does not act as a mutex. Branch protection controls merges, not deployment ordering. The race happened because two approved deploy jobs ran concurrently. Combine controls: use a workflow-level or job-level concurrency group for the environment, set cancel-in-progress: false for production to queue, and add an environment deployment branch policy limiting deploys to main. Use required reviewers for human authorization, but do not treat them as serialization. If you need true latest-wins, use cancel-in-progress: true only after making the deploy idempotent and adding a post-deploy digest check. Also ensure the smoke test uses the deployment's immutable image digest, not the service name, and surface the deployed revision in the environment URL.