The Bucket That Wasn't Public
A GuardDuty finding flags an IAM access key used from an unfamiliar ASN, followed by 12,000 S3 GetObject calls in four minutes against the bucket holding customer statements. The key belongs to a CI/CD service account whose policy is AdministratorAccess; it was stored as a Jenkins credential, and two developers hold admin rights on that Jenkins instance. Block Public Access is on, and the bucket policy allows s3:GetObject to any principal conditioned on aws:SourceVpc matching the CI VPC. CloudTrail is enabled in us-east-1 only; the organisation runs workloads in three regions; S3 data events are not enabled anywhere. The finding arrived 40 minutes after the first GetObject.
1. How confident can you be about what was actually taken, and what do you do about that uncertainty?
Read the worked answer
You cannot be confident, and you should say so in those words.
- S3 data events are off, so there is no object-level record of which keys were read. You have a GuardDuty volume anomaly, not a list of exfiltrated objects.
- CloudTrail is single-region. GuardDuty is multi-region by default, which is why the finding surfaced, but if the bucket is in another region the management events and any future data events are not where you are looking. Verify the bucket's actual region first - this is a five-minute check that determines whether you have any logs at all.
- The 40-minute lag means the flow may still be running or may have finished; GuardDuty timing is not incident timing.
What to do with the uncertainty
- Bound the exposure by content, not by key count: which policy or naming convention governs 'customer statements,' and how many objects would 12,000 GETs plausibly touch in four minutes? Estimate bytes-per-object and compare with VPC flow logs / S3 server access logs if any exist.
- Turn on data events now for that bucket so the next four minutes are visible, and enable org-wide trail. Post-incident this is mandatory; during the incident it protects the remaining window.
- Assume disclosure for the data class in the bucket until proven otherwise - notification clocks are legal, not technical, and 'we couldn't tell' does not stop them.
Judgement The honest answer to 'how much was taken' is a bounded worst case, not a number. Build the notification decision on the bucket's contents, not on the logs you wish you had.
2. Give your containment sequence, including what you must not break.
Read the worked answer
Order by what the attacker can escalate to next, not by what is easiest to click.
- Disable the access key and its session tokens. Immediate and reversible. But first check what else that identity runs: a CI/CD account is often the deployment path, so disabling it can halt all releases. Pre-create a replacement identity with least privilege and a scoped credential before you disable, or you trade a breach for an outage.
- Revoke Jenkins' stored credential and rotate everything the account could reach. AdministratorAccess means you must assume it read secrets, SSM parameters, and other credentials - rotation is not optional.
- Contain the Jenkins instance. Two developers with admin rights plus a leaked credential means the credential store, job configs, and build logs are all suspect. Snapshot it for forensics before cleanup; do not just rotate the credential and move on.
- Then review for persistence: new IAM users/roles, trust policy edits, Lambda functions, and any change in the other two regions.
Must not break Deployment pipeline (step 1 handles it), and the immutable/versioning state of the bucket - do not delete versions or change the policy in a way that destroys the evidence of what was written.
Judgement Rotating the key alone is the classic incomplete containment: the adversary had AdministratorAccess, so every secret that identity could read is now compromised, including the ones that would let them come back without the key.
3. Explain why the aws:SourceVpc condition provided no protection, and what the correct design is.
Read the worked answer
The condition was attached to an Allow, and an Allow cannot subtract permissions.
Two independent reasons the bucket was exposed:
- Same-account identity policies win. The CI/CD role has AdministratorAccess, i.e.
s3:*on*in an identity-based policy. For a principal in the same account, S3 access is granted by either the identity policy or the bucket policy. The bucket policy was never needed, so its SourceVpc condition was never evaluated as a restriction. Only an explicit Deny - in the bucket policy, an SCP, or a permissions boundary - can take permissions away from an identity policy. aws:SourceVpcis only present for requests through a VPC endpoint. A request from the internet has no such key, so the condition is false and the Allow simply doesn't apply - which is safe only if there is no other grant. Here there was another grant.
Correct design
- Explicit Deny on the bucket:
Deny s3:*whenaws:SourceVpc != vpc-xxxx, with the standardaws:SourceVpcnull-check caveat handled, plus a second Deny fors3:GetObjectoutside the CI role. - Remove AdministratorAccess from the CI role. Nothing in a build pipeline needs it; scope to the specific buckets and actions.
- Add an SCP that denies broad S3 actions org-wide as a backstop.
- Enable S3 data events and an org-wide CloudTrail.
Judgement The team believed they had a network restriction and actually had a comment. Defence in depth on object storage requires a Deny somewhere in the chain plus least-privilege identity policies; either one alone fails this way.