74 AWS accounts, 11,400 findings, and one leaked access key

The team did not fail at fixing - it failed at ranking, and the evidence is the abandoned meeting, which is what happens when a list has no actionable ordering. A severity label from a scanner is not a priority.

Reorder by exploitability × blast radius × reachability, then fix at the source:

1. Internet-reachable + data-bearing. Public S3 buckets with claims data, unrestricted security groups on 22/3389 on instances with public IPs, public RDS snapshots, exposed EKS API endpoints without private access. These are the finding types that appear in the Cloudpost incident's family.
2. Identity and key material. Long-lived access keys (count them: how many exist, how old, who owns them), roles with wildcard actions on wildcard resources, and - critically - which OUs lack permission boundaries. The incident showed the boundary was an accident of OU placement, not a design.
3. Audit and detection integrity. CloudTrail configuration per account and region, log archive account, SCPs protecting them. Everything downstream depends on this, so it goes early even though scanners rate it medium.
4. Everything else - encryption-at-rest defaults, minor version drift, tagging. Batch, fix via IaC modules, do not track individually.

Practical mechanics: compute the top 20 fixes that close the largest share of genuinely risky findings and assign named owners per account. Set a rule that no new finding of categories 1-3 can be created from this point forward (preventive guardrails at deploy time), so the backlog is a decreasing problem rather than a refilling one. Report backlog trend by category, not total count. And reinstate the meeting only when there is a short list a human can act on - a meeting about 11,400 items deserves to be abandoned.

Assume the attacker achieved persistence and that deleting the visible user removes only the noisiest artifact. The visible IAM user is the part they expected you to find.

Investigation checklist, ordered by how long the artifact survives cleanup:

1. IAM role trust policies. An attacker can create a role trusting an external account or an OIDC provider they control. That survives deletion of every user and key. Enumerate every role with a trust policy referencing a principal outside your org, and every role created or modified in the window.
2. CloudTrail tampering. One region's trail was modified - enumerate every region for each account, verify trails exist, verify they deliver, and check for StopLogging, UpdateTrail, DeleteTrail, and EventBridge rule deletions. Treat the missing region as a coverage hole and re-derive activity there from VPC flow logs and S3 access logs.
3. Other credential material reachable from the compromised identity. SSM parameters, Secrets Manager entries, S3 objects with config files, EKS secrets, EC2 instance profiles and user data. If the key had read access to these, rotate them all.
4. Resource creation outside normal regions. 22 GPU instances were obvious; quiet persistence lives in unused regions - Lambda functions, EventBridge scheduled rules, and KMS grants.
5. Data access, not just resource creation. The spend figure is visible; S3 GETs against the claims buckets may not be. Pull data-event logs if enabled, and if they are not enabled, say so explicitly - that is a finding.
6. The CI/CD path. 220 deploys a week means a compromise in the pipeline can re-create infrastructure after cleanup. Audit recent pipeline runs and the credentials they used.

Then rotate everything the identity could reach, not just what it demonstrably used. The judgement being tested: containment for a leaked cloud credential is broad and fast, and scoping is done afterward from logs - never before.

Map controls to each step of the chain, then test them.

Prevention:

• Key never existed / never leaked: replace long-lived keys with OIDC federation from CI, and require human access via SSO with short-lived credentials. Add pre-commit secret scanning plus push protection at the repository - this is the specific control that fails in the incident, and server-side push protection matters because client-side hooks are bypassed.
• Blast radius: permission boundaries on every account, not just the prod OU, applied via SCP and enforced at role creation. The incident's AdministratorAccess user would have been capped.
• Audit integrity: SCPs denying cloudtrail:StopLogging, DeleteTrail, and region disablement across all accounts, plus an organization-wide trail with a log-archive account the workload accounts cannot write to.
• Region governance: an SCP denying resource creation outside approved regions, which makes the "quiet region" persistence trick expensive.

Detection (runtime, not posture):

• EventBridge or CloudTrail-based rules on CreateUser, CreateAccessKey, AttachUserPolicy, UpdateAssumeRolePolicy, and trail modification, routed to an on-call channel with an automated response that disables the principal and notifies the owner. The absence of this detection is why a 14-hour window produced $31,000.
• GuardDuty and anomaly detection on spend and on new-region API activity.

Verification - the part usually skipped: Run a quarterly canary exercise. Place a decoy access key in an internal repository, confirm push protection blocks it; if bypassed, confirm the detection fires and the automated response disables the key within minutes. Separately, attempt CreateUser in a non-prod account with the canary identity and prove the boundary and SCP both deny it. Log the exercise result as a control owner signature. A control that has never been tested against the exact attack you experienced is a hypothesis, and the exam expects you to say so out loud.