Containing a Leaked CI/CD Key in a Multi-Account Landing Zone
Helios Pay runs a 14-account AWS landing zone under AWS Organizations. A GitHub Actions workflow deploys to a sandbox account using an IAM user access key stored as a repository secret. The key has AdministratorAccess because the team could not make cross-account roles work before a release. GuardDuty reports cryptocurrency mining on three EC2 instances in the sandbox; CloudTrail shows the key creating IAM users and modifying security groups. The organization has an SCP that denies regions outside us-east-1 and eu-west-1, but it is attached only to the production OU. The security team wants to delete the key immediately, disable the sandbox account, and block GitHub. The platform team worries this will stop all deployments.
1. What containment order do you recommend in the first hour, and why not simply delete the key and disable the account immediately?
Read the worked answer
Contain in this order: disable the access key and revoke active sessions, isolate the three EC2 instances with a forensic security group that allows no outbound except to your evidence host, snapshot EBS volumes, and preserve CloudTrail and VPC Flow Logs. Then attach an SCP to the sandbox OU denying new IAM users and expensive resources. Do not delete the key first; disabling it stops use while preserving the key ID and CloudTrail history for investigation. Do not disable the whole account until logs and snapshots are secured, because account suspension can interrupt access to evidence and complicate recovery. Blocking GitHub entirely is too broad; instead, disable the workflow that uses the key and rotate any other secrets in that repository. Notify the platform team with a specific exception path so releases can resume through a reviewed role after containment.
2. Redesign GitHub Actions access to the sandbox and production with least privilege. Include trust, permissions, and guardrails.
Read the worked answer
Use GitHub OIDC federation instead of long-lived keys. Create an IAM OIDC identity provider and one role per account and environment. The trust policy should restrict token.actions.githubusercontent.com:sub to a specific org/repo and environment, such as repo:heliospay/app:environment:prod, not a wildcard. The role should allow only the deployment actions needed, with a permissions boundary, short maximum session duration, and separate plan and apply roles. Production apply should require a GitHub environment approval and should not allow IAM user creation. Add SCPs to every OU, not only production, denying IAM users, root access keys, and unapproved regions. Centralize CloudTrail and GuardDuty findings, enable secret scanning, and remove repository secrets that contain AWS credentials.
3. Which controls would have detected or limited this incident earlier?
Read the worked answer
The key had AdministratorAccess and the SCP covered only production, so the sandbox was effectively unguarded. Controls that would have limited impact include organization-wide SCPs denying IAM user creation and restricting regions, IAM Access Analyzer findings for external access, and permission boundaries on CI roles. Detection would improve with GuardDuty, CloudTrail anomaly alerts, billing alarms for sudden EC2 spend, and GitHub secret scanning with push protection. A central log archive account with immutable retention would preserve evidence. The platform team should also monitor for security group changes to 0.0.0.0/0 and for new IAM users. These controls do not replace the OIDC redesign, but they shorten the window between key exposure and containment.