Ransomware Recovery When the Attacker Held Admin for Two Weeks
An attacker using stolen administrator credentials spent two weeks inside a production AWS account running a 200-instance order platform. On detection, the on-call engineer terminated every instance and deleted the access keys; the platform was serving again in 40 minutes. The investigation later found that EBS snapshots in the same account had been deleted, and that the attacker had also reached the replication account in a second region, where 30-day backups were stored. Compliance requires a 36-hour RTO for order processing, and legal wants forensic images of the affected hosts. The regulator requires notification within 72 hours of confirming a personal data breach; the SIEM retains 14 days of logs.
1. Critique the response team's order of operations. What should the first actions have been?
Read the worked answer
Terminating instances is a destructive containment action taken before containment was actually achieved, and it destroyed the evidence that would have answered the two questions that mattered: what did they touch, and did they take data.
The first actions should have been ordered by what is reversible:
- Revoke and rotate the credential and its sessions - the instances were not the attacker's access; the identity was. Termination without revocation changes nothing about their reach.
- Quarantine, don't destroy. Move instances behind a deny-all security group and detach from load balancers; keep them running and reachable only from the forensics subnet.
- Snapshot EBS volumes and, where tooling exists, capture memory before anything is rebuilt. Memory holds the injected tooling and often the only record of what was exfiltrated.
- Rebuild into a known-good environment from infrastructure-as-code once the credential path is closed.
The 40-minute recovery is genuinely good for availability and genuinely bad for the investigation. Both are legitimate objectives; the mistake is that nobody sequenced them.
2. The backups were in a second region and were still compromised. What does that tell you about the backup design?
Read the worked answer
That the design confused geography with isolation. Cross-region replication protects against a regional outage; it does nothing against an identity that can reach both regions. The attacker did not need to defeat the backup technology - they simply used the same credentials.
What the design needs:
- A separate recovery account with a distinct credential plane, no trust relationship from the production identity, and ideally a separate organisation or at minimum a hard organisational boundary.
- Immutability - object lock or vault lock with a retention period in compliance mode, so even a legitimate admin cannot delete within the window. Deletable backups are copies, not backups.
- Restore credentials not usable for ordinary operations, and not present in the production environment at all.
- Monitored, periodic restore tests, because an untested vault has unknown recovery time.
- Deletion alerting on the backup account, since mass deletion of snapshots is a detectable precursor hour before encryption.
The secondary lesson is that the 30-day retention meant nothing because the deletion happened at day 14 with no recovery point behind it.
3. Can the 36-hour RTO and the 72-hour notification both be met, and what would you tell the business?
Read the worked answer
The RTO is plausible; the notification is the constrained one, and the business should hear both plainly.
RTO. Rebuilding 200 instances from infrastructure-as-code is hours, not days - if the dependencies are recoverable. The realistic risks are not compute: deleting a customer-managed KMS key renders every snapshot permanently unreadable; secrets manager entries, DNS zones, and the IAM structure all have to be restored in the right order. The 36 hours should be modelled as a dependency chain, and the recovery plan must include key material and secrets, not just instances.
Notification. Fourteen days of SIEM retention against a two-week dwell time means the earliest days of the intrusion are already unobservable, and logs in the compromised account are suspect anyway. You may not be able to determine the scope of any personal data exposure within 72 hours.
The honest message is: commit to the 36-hour RTO with a tested dependency map, and prepare a staged breach notification with a documented, defensible basis for preliminary scope rather than promising certainty the telemetry cannot support. Recommend extending log retention to at least 12 months in a separate, write-once account.