Removing Long-Lived Secrets from 120 Pipelines

Replace the stored key with a workload-issued identity token.

Mechanism: configure the JWT authenticator against the CI provider's OIDC issuer. At run time the pipeline requests a short-lived OIDC token (the job must be granted permission to request one), presents it to the authenticator endpoint, and receives a short-lived Secrets Manager access token in exchange. The pipeline then retrieves only the variables its identity is entitled to. Nothing is written to disk, and ephemeral agents need no cleanup because the token dies with the job.

Scoping decisions that matter:

• One host identity per repository and environment, not one per organisation. payments-app/production and payments-app/staging are separate identities with separate entitlements, so a staging job cannot read production values.
• Map identity from a claim, not from a shared secret. Bind the authenticator to the issuer and use the token's subject and repository claims to resolve the host identity, so an operator cannot rename a job to gain another team's identity.
• Restrict which claims are acceptable. Configure the authenticator to reject tokens whose issuer, audience, or repository does not match the expected values. This is the single most important control: an authenticator that accepts any token from the provider lets any repository in the organisation assume any mapped identity.
• Grant at layer level, narrowly. Each host goes into a layer whose entitlements cover exactly its variables. Nothing is granted branch-wide.

Verify with a negative test: run a job in a repository that should not have access and confirm authentication or retrieval fails.

The shared issuer is not the control; the claim restrictions and the entitlement model are.

Three layers of protection, all required:

1. Authenticator claim enforcement. Constrain the JWT authenticator with enforced claims so a token must carry the expected repository, ref, and environment values before it can map to any host identity. Without this you have one trust relationship for the whole provider account, which is a single point of over-privilege.
2. Per-identity entitlements. The payments host lives in the payments layer, which is granted the migration credential and nothing else. Even a valid payments token cannot retrieve loyalty variables because no grant exists. Entitlements are the actual authorisation boundary; claim restrictions only decide who you are.
3. Pipeline-definition control. Anyone who can edit the workflow file can ask for a token and retrieve whatever that identity is entitled to. So restrict who can change pipeline definitions, require review, pin allowed refs and environments, and put the production step behind a protected environment with required approvers. A contractor with repository write access is otherwise inside the trust boundary.

Test it: a job on a feature branch in the payments repository should fail if only the release branch is permitted; a job in an unrelated repository should fail at authentication; a payments job on the release branch should succeed for the migration credential and fail for anything else. Record those three results as evidence - the auditor will ask how you know the boundary holds, not whether you configured it.

Treat this as a release with a parallel run, not a switch.

Preparation:

• Create the payments release host identity, its layer, and a grant for the migration credential only. Use the database's own short-lived credential mechanism if it supports one, so the job receives a credential that expires with the run rather than a static password.
• Put the production step behind a protected environment with required reviewers, so the OIDC token can only be minted after approval.
• Confirm the database account used for migrations has only DDL rights on the target schema and no standing read access to production data.

Cutover:

1. Run the new pipeline on a non-production environment and compare the resulting schema with what the Jenkins job produces.
2. Run both pipelines against production for one release, with only the new one applying changes and the Jenkins job held in dry-run, so failures surface without double application.
3. For two subsequent releases, run only the new pipeline. Compare timings and failure modes.

Rollback: keep the Jenkins credential in place and the job definition intact until two clean releases have passed; rollback is re-enabling the Jenkins path and revoking the new host identity. After the hold period, delete the Jenkins credential, scan the repository and job configurations for references, and confirm zero authentications from the old identity in audit records before removing it from policy.

Timing note: migrations that lock tables are production-impacting. Schedule the cutover where a failed run can be completed by hand in the same window, and keep the old credential available through at least one such window.