Long-lived CI/CD API keys after a contractor departure
A platform team supports 35 CI/CD pipelines on IBM Cloud. Each pipeline authenticates with a service ID API key stored as a toolchain environment variable. There are 52 service IDs; 19 keys are older than 400 days. One key belongs to a shared automation-admin service ID with Administrator on all resource groups. A contractor with toolchain access left two weeks ago, and Activity Tracker shows the automation-admin key was used from an unfamiliar IP address to list COS buckets and read one bucket's objects. The team wants to keep releases running while removing long-lived credentials. They have IBM Cloud Secrets Manager, IAM access groups, and context-based restrictions, but no enterprise OIDC integration yet. The security lead asks for containment, a target pipeline identity design, and policy guardrails that will not break the 35 pipelines at once.
1. What are your first containment and investigation steps, and in what order?
Read the worked answer
Containment starts with the shared automation-admin key because it has the broadest privilege and confirmed suspicious use. Disable or revoke that API key first, but preserve its metadata and recent Activity Tracker events before deleting anything. Revoke the departed contractor's identity and any tokens or toolchain access still associated with it. Then search Activity Tracker across IAM, COS, Key Protect, and account settings for the same service ID and source IP: new API keys, new IAM policies, new service IDs, bucket policy changes, key state changes, and other data reads. Check COS access logs or activity events for the scope of the bucket read.
Do not rotate all 52 service IDs blindly in the first hour; that may break releases and obscure evidence. Instead, isolate the confirmed compromised identity and any identities the contractor could reach. Rotate the 19 keys older than 400 days in a controlled wave after confirming which pipelines own them. Move new keys into Secrets Manager and update pipelines one at a time.
The investigation should answer three questions: what data was accessed, whether the actor created persistence, and whether other service IDs were used from the same source. If persistence is found, remove it before restoring normal releases. Preserve logs in a separate evidence location with restricted access.
2. Design the target pipeline identity model. How do you reduce long-lived credentials without an enterprise OIDC integration yet?
Read the worked answer
The target should be one workload identity per pipeline and environment, with no shared automation-admin identity. Create separate service IDs for production and non-production, attach them to access groups, and grant only the resource group and service roles each pipeline needs. A pipeline that publishes to one COS bucket should not have account-level Administrator or broad Key Protect management rights. Where possible, separate build, deploy, and runtime identities so a compromised build step cannot deploy to production.
Without enterprise OIDC, the practical interim step is short-lived, rotated API keys stored in Secrets Manager, with automated rotation and pipeline retrieval at runtime. Set a maximum key age, alert before expiry, and remove keys from toolchain environment variables. Use context-based restrictions to limit where those keys can be used, such as approved toolchain egress addresses or a private network path. This does not eliminate the secret, but it reduces exposure time and makes theft from one location less useful.
Plan the migration to trusted, federated workload identity as the strategic endpoint. Until then, treat API keys as transitional credentials with owners, rotation dates, and inventory. The design should make the shared admin identity impossible to recreate by policy and review, not just by convention.
3. Which IAM and context-based restriction guardrails would you put in place, and how would you roll them out safely?
Read the worked answer
I would start with deny-by-default at the account edge for human administration: no standing account-level Administrator except a break-glass access group, MFA required for human users, and access reviews for every access group that can change IAM or key state. For workloads, prohibit service IDs with account-level Administrator; require resource-group-scoped roles and service-specific roles. Add alerting on any new policy that grants Administrator, any new API key for a privileged service ID, and any access-group membership change.
Context-based restrictions should be scoped to the sensitive services first: COS, Key Protect, and IAM. Build an allowlist from observed pipeline egress addresses and private endpoints, run in a monitored or audit phase, and review denials before enforcement. Do not enforce a broad rule that locks out the toolchain during a release window. Roll out by pipeline group, starting with non-production, then production.
Secrets Manager should hold the new keys, with rotation ownership recorded. Finally, add a periodic reconciliation: every service ID must have an owner, a purpose, a last-used date, and a maximum key age. Guardrails are only effective if exceptions are visible and time-bound; otherwise the shared admin identity reappears under a different name.