Leaked Access Key, 2.1 TB OBS Egress: Containment and Redesign

1. Preserve evidence first. Do not delete the bucket or objects. Capture the request IDs, source range, timestamps and object list from OBS logs, and copy the current CTS traces somewhere safe before anything else changes.
2. Revoke the credential, not the password. Disable or delete the access key of svc-report (or the user itself). Changing an IAM user password does not invalidate an existing AK/SK - this is the single most common containment mistake.
3. Detach OBSFullAccess from the user; replace with a deny-by-default stance temporarily.
4. Block the source with an explicit Deny on s3:* for the offending IP range in the bucket policy. In OBS semantics an explicit Deny overrides any Allow, so it takes effect even while other grants still exist.
5. Audit for persistence: check for newly created IAM users, access keys, agencies, modified bucket ACLs, new CTS trackers, changed SMN subscriptions, and any other AK older than the breach window. Delete the root access key outright - it is never required for daily operations.
6. Enforce MFA on root and on all 12 IAM users; enable login protection.
7. Scope the breach: which objects left, do they contain personal data? That answer drives the legal/regulatory notification clock (for example, GDPR-style 72-hour reporting). The CTO's question cannot be answered until the object list is known.

The flaw: the audit trail lives inside the resource being audited, and it is protected by the same credential that was compromised. An attacker with OBS read - or worse, write - permission can read the traces to learn what is monitored, and can delete or overwrite them to erase the record of the exfiltration. In this incident the thief could have been reading the very log of their own activity.

Correct structure:

• A dedicated log bucket, ideally in a separate account under an organization, with a bucket policy that denies DeleteObject to everyone including the account administrator, and WORM/retention enabled for the compliance period.
• The CTS tracker on the production account delivers to that bucket only.
• OBS access logging enabled on the invoice bucket, also writing to the log account.
• SMN notifications on the tracker for high-risk events (AK creation, policy changes, bucket policy changes) so detection does not depend on someone reading logs later.
• Keep log retention longer than the longest plausible investigation window - for invoice data, at least one year.

Principle: audit data must be in a different failure and trust domain from the data it describes.

Stop using permanent AK/SK for workloads.

• Any application on ECS should obtain temporary credentials from an IAM agency attached to the instance, so nothing is written into code or Git. Credentials rotate automatically and expire.
• If an external system genuinely needs a key, store it in a secrets manager, rotate on a schedule (≤90 days), and scan repositories in CI for key patterns with pre-commit hooks.

Least privilege:

• Delete the OBSFullAccess assignment. OBSFullAccess lets a compromised key delete every bucket in the account; it is never appropriate for a service account.
• Grant action-scoped, prefix-scoped permissions: GetObject on invoices/2024/* only, no DeleteObject, no PutBucketPolicy.
• Layer a bucket policy with conditions (TLS required, allowed source VPC endpoint) so an IAM misconfiguration alone is not sufficient.

Defence in depth:

• Enable versioning + WORM so mass deletion cannot destroy data, and SSE-KMS so objects are useless if copied out without the key.
• Enable HSS on hosts and CES alarms on OBS egress volume and request-rate anomalies - this incident was caught at 02:40 only because egress crossed a threshold; add a request-rate alarm on 4xx/403 spikes, which usually precede a successful key abuse.
• Separate production, logging and audit accounts, and apply Enterprise Project scoping so one team's mistake stays inside its boundary.