Defensible evidence from a compromised EC2 instance inside an Auto Scaling group

First, stop the clock on automated destruction. The ASG is the biggest threat to the evidence: a health-check failure terminates the instance, and with default settings the root volume is deleted with it.

1. Suspend the ASG's terminate/replace processes (ReplaceUnhealthy, Terminate, Launch) and put the instance into Standby - in service terms it is removed from the load balancer, in evidence terms it stays alive. Record the API calls and timings.
2. Quarantine the network. Swap the security group to one with no egress except to the forensic tooling path; adjust NACLs if the SG alone leaves lateral paths. Do not stop the instance - stopping loses memory, and starting again may run the miner's persistence.
3. Defang the identity. Strip or neuter the instance profile so it cannot continue reading the invoice bucket; check S3 server access logs and CloudTrail data events immediately for prior GetObject calls from that role session.
4. Capture memory before anything else touches the host. SSM Run Command is the only channel you have; run the acquisition tool writing to a separately attached EBS volume, not the root volume, and record the command ID, parameters, timestamps and role used. Accept that this is live response - the instance's memory is already altered by the mining process - and document that.
5. Snapshot the root volume and any attached data volumes, tagged with the case ID. Note that EBS snapshots are crash-consistent: no memory, no network state, no open file handles.
6. Preserve logs - the S3 CloudTrail trail is authoritative over the 90-day CloudWatch copy; also export VPC Flow Logs, ALB access logs, GuardDuty findings with their supporting data, and S3 access logs. Copy them into an evidence bucket with Object Lock.
7. Record instance metadata (instance ID, ENIs, private IP, key pair, launch time, AMI, IAM role) as a structured export before anything changes state.

Only after that do you analyse - and on a copy restored in an isolated forensic account, never on the source.

Your chain of custody is the API log. There is no signed evidence bag, so the tamper-evident record is CloudTrail: every snapshot, every SSM command, every GetObject, tied to a role session and a timestamp. Preserve your own actions as part of the case file - they prove what was done, by whom, from which account, and in what order.

Practical requirements:

• Unique case ID on every resource (snapshot tags, bucket prefix, S3 object metadata) linking objects to the case.
• A written evidence log in UTC: action, operator, credential used, API call, resulting resource ID, hash.
• Hashing: you cannot hash an EBS snapshot object directly. The defensible method is to create a volume from the snapshot in an isolated forensic account, read it block-by-block, and compute SHA-256 over that volume, documenting that the hash is of a derived volume, not of the snapshot. Alternatively use the EBS direct APIs to read and hash blocks, recording the byte ranges. State the method explicitly in the report - an unexplained hash invites challenge.
• Memory image: hash immediately after acquisition on the instance, then re-hash after transfer; record both values and any discrepancy.
• Storage: copy snapshots to a dedicated forensic account, re-encrypt with a forensic KMS key, restrict access to a named case role, and place logs in an Object Lock bucket in compliance mode. Keep an untouched master copy and analyse only working copies.
• Clock discipline: CloudTrail is UTC and authoritative; the host clock may be skewed and the application logs to stdout via CloudWatch with ingestion timestamps, not event timestamps. Record the offset and build the timeline on CloudTrail where possible.

Document the tools and versions used for acquisition and analysis. Where the instance was already compromised, state plainly that live response alters state - the record of how it was altered is what keeps the evidence defensible.

1. Letting the ASG terminate the instance. The single most common cloud forensics failure: the instance is replaced, the root volume is deleted (default DeleteOnTermination=true), and the only copy of the malware is gone. Avoidance: suspend the ASG processes and move the instance to Standby before investigating anything, and verify DeleteOnTermination on the launch template so you know the risk.

2. Running investigation commands on the compromised host. A host with a miner and unknown other implants may have tampered tooling, hooked binaries, or a rootkit that will lie to you. Running ps, netstat and the like through SSM produces output that is neither complete nor trustworthy, and it changes memory. Avoidance: capture memory first with a known-good tool from your own volume, then stop interacting; do all analysis on restored copies.

3. Untracked SSM output. Run Command writes output to S3, where anyone with bucket write access can alter it; it is investigator-generated, not host-generated. Avoidance: restrict the bucket, enable versioning and Object Lock, and record command IDs and result hashes in the case log.

4. Unstated acquisition artefacts. Failing to disclose that snapshots are crash-consistent, or that the memory image was taken after the miner had been running for hours, lets the other side argue the evidence is not representative. Avoidance: a method section that states limitations openly - disclosed limitations survive cross-examination; discovered ones do not.

5. Identity ambiguity. The instance role can be exercised from anywhere; CloudTrail shows the role session, not the process. Avoidance: correlate with VPC Flow Logs, ALB logs, and any process telemetry to establish which human or implant made the call, and say so when you cannot.

6. Relying on the GuardDuty finding as evidence. It is a detection signal, not proof. Export the finding and the raw data behind it, and build the case from primary logs. Finally, confirm the legal hold covers all accounts with relevant data - the invoice bucket's S3 access logs may sit in a different account with a shorter retention than CloudTrail's.