A Cryptomining Detection That Waited Five Hours for Break-Glass Approval
At 02:40, a detection fires in a production workload account: compute resources are contacting a mining pool. The on-call security engineer has read-only access to that account. Containment requires isolating the instance and revoking credentials, which needs the break-glass role; policy requires approval from a manager and the cloud platform lead, both offline until 08:00. By the time approval arrives, the attacker has terminated the instance and its attached volume, and the workload account's local log retention is 90 days but the relevant trail entries are gone. The organization-level trail in a separate log-archive account still holds the API history, and object lock is enabled on the archive bucket. The organization runs 380 accounts. The post-incident review finds this was the third time in a year that containment waited on approvals, and that the break-glass role has never been tested.
1. Reconstruct what should have happened between 02:40 and 08:00 and identify the specific process and design failures.
Read the worked answer
At 02:40 the alert should have opened an incident automatically, paged a second responder, and started a timeline. The first responder's job in the first fifteen minutes is containment and preservation, in that order: isolate the instance by replacing its security group with a deny-all quarantine group, snapshot the attached volume, capture instance metadata and running processes if an agent permits, then revoke the instance's credentials and any session tokens, then disable any associated access keys. Do not terminate the instance; termination destroys memory and often the disk with it.
The failures are specific and separable.
Process failure: containment authority depended on two humans being awake. Approval latency was unbounded and there was no documented out-of-hours approver.
Design failure: the on-call role had read-only access in the affected account, which means the person expected to contain an incident could not act. The break-glass role existed but had never been tested, so nobody knew whether it worked, which is likely part of why it was not used.
Evidence failure: the account-local trail was insufficient and the 90-day local retention was irrelevant once the instance was deleted, because the attacker removed the resource, not the log. The organization trail survived only because it was segregated; that segregation was the one thing that worked.
Detection-to-containment took roughly five hours and twenty minutes against an adversary who needed minutes.
2. Redesign the authorization model for incident containment so it is fast but still accountable. What are the tradeoffs?
Read the worked answer
The goal is not fewer approvals; it is approvals that do not depend on a specific person being awake.
Design: a scoped containment role that can do a narrow, enumerated set of actions across the 380 accounts, namely quarantine an instance, revoke credentials, disable an access key, snapshot a volume, and block an identity. It cannot delete resources, change billing, or alter IAM policy. It is eligible to the on-call security engineer for the duration of their shift and is activated with step-up authentication.
Accountability replaces pre-approval with post-hoc review: activation requires an incident ticket ID, every action is logged with the actor and the justification, a second responder is paged automatically at activation, and a security lead reviews all activations within 24 hours. Two-person integrity can be preserved for destructive actions such as key deletion or account suspension, while containment stays single-approver.
Tradeoffs to name explicitly. Broader standing privilege increases the damage a compromised on-call account can do, which is why the role must be narrowly enumerated, session-bound, and monitored for use outside an open incident. Post-hoc review means a bad containment decision may stand for hours; mitigate with a second responder and a clear rollback path for quarantine actions, which are reversible. Automated approval for destructive actions would be faster still, but the reversibility asymmetry does not justify it.
Finally, test it. Quarterly game days that activate break-glass, contain a synthetic incident, and measure time to containment. An untested role is an assumption, not a control.
3. What evidence gaps exist in this incident now, and how would you change logging and retention across 380 accounts to close them?
Read the worked answer
Current gaps: no disk or memory image, because the instance was terminated; no process tree or network connections from the host; the account-level trail entries are missing, so the local view of what happened in that account is incomplete; no record of what the mining process executed or what it downloaded. What survives is the organization-level API history, which tells you which identities called which APIs, from where, and when. That is enough to scope credential misuse and to find persistence, but not enough to understand the payload.
Changes:
Centralize and segregate by default. Every account ships its trail to the log-archive account in the organization, with a resource policy that denies deletion and modification even to account administrators. Object lock with a compliance-mode retention period covers the audit window. Verify that no account can opt out and alert on any attempt to stop delivery.
Shorten local retention. Ninety days of local logs invites the belief that evidence is safe locally. Keep seven to fourteen days locally for troubleshooting and treat the archive as the system of record.
Automate preservation. On incident declaration, the response automation should snapshot attached volumes, export relevant log slices to an evidence bucket with a hash manifest, and tag the instance as evidence so nothing deletes it.
Detect tampering. Alert on trail modification, on snapshot deletion, and on volume deletion in accounts with an open incident. The attacker in this case destroyed evidence; that behaviour should itself be a detection.
Rehearse retrieval. Quarterly, pick a random past date and prove the archive can return a complete, admissible record for a given account within the incident SLA.