Least Privilege for a Shared CI Pipeline
A company runs a CI/CD pipeline on a build server that deploys a containerized application to production. Today the build server holds a long-lived access key for an IAM user with AdministratorAccess, because "it was the only thing that worked" during an incident two years ago. The security team found the key committed in a private repository's history. Three teams share the same build server, and each team deploys to its own environment. Auditors require that no long-lived credentials exist in the pipeline, that each team can deploy only to its own environment, and that deployments be attributable to a person or pipeline run. The engineering manager worries that tightening permissions will break the nightly deploy and stall releases. The company already uses a cloud provider's identity federation for employees and has a ticketing system that records approvals.
1. Design the credential and permission model that satisfies the auditors without blocking releases.
Read the worked answer
Replace the long-lived key with short-lived credentials issued to the pipeline itself. The build server (or, better, a managed CI runner) assumes a role through workload identity federation or an instance profile, so no static secret exists in the repository or on disk. Create one deployment role per team-environment pair, each scoped to the specific resources that team owns (its container registry path, its deployment service, its target environment) rather than AdministratorAccess. Use role session names and tags that encode the team, environment, and pipeline run ID so every action is attributable. The nightly deploy keeps working because the runner's identity is stable and the role trust policy allows it; releases are not gated by human approval unless policy requires it. For human-initiated production deploys, require an approval recorded in the ticketing system before the pipeline assumes the production role. Rotate and revoke the exposed key immediately, purge it from history, and add secret scanning to block recurrence. The manager's fear is addressed by rolling out in report-only mode first: run the narrowed policies in a non-blocking audit mode, fix gaps, then enforce.
2. What should be done about the exposed key, and in what order?
Read the worked answer
Order matters because containment precedes cleanup. First, deactivate or delete the exposed access key so it cannot be used, and check logs for any use from unexpected locations or times; if there is evidence of misuse, treat it as an incident and escalate. Second, confirm the pipeline still deploys using the new short-lived credential path before removing the old one, so you do not trade a security fix for an outage. Third, remove the secret from the repository history or, if that is impractical, treat the secret as permanently compromised and rely on revocation rather than deletion. Fourth, add preventive controls: secret scanning with push protection, a policy that blocks long-lived keys for pipeline identities, and alerts on creation of new access keys. Finally, review why the broad role was created during the incident, and document an emergency-access process that grants temporary elevated permissions with automatic expiry and logging instead of a permanent AdministratorAccess key. The lesson to encode is that break-glass access should be time-bound and auditable.