02:10 Saturday: Three Servers Encrypting
At 02:10 on a Saturday, the on-call analyst sees rapid file-rename activity on three of forty application servers, consistent with ransomware. The environment is one domain with a flat internal network, and the backup repository is joined to that same domain. The most recent verified full backup is 31 hours old; nobody has run a restore test since the platform was rebuilt eight months ago. The plant operations director says halting the production line costs roughly 180,000 dollars an hour and asks you to keep the three servers running while you investigate. The incident response retainer guarantees a responder within four hours. No external notification has been made.
1. Isolate now or investigate first? Make the call and name the evidence you are trading away.
Read the worked answer
Contain now - but contain at the network layer, not the power switch.
The decision rule is comparability of costs. Three servers already show destructive activity, which means the perimeter is not holding; the cost of containment is bounded and calculable in hours of production, while the cost of uncontrolled spread across a flat network of forty servers is not bounded at all. When one cost is bounded and the other is not, you pay the bounded one.
Before the hosts, protect the backup repository. Take it off-domain and rotate its credentials first. Everything else on this list is recoverable; a domain-joined backup repository that is encrypted or held is not.
Evidence consciously traded: live attacker interaction and attribution, plus volatile memory if hosts are suspended rather than isolated.
How to reduce the trade: isolate at the switch or by VLAN move so hosts stay powered and memory survives, and capture a memory image and disk snapshot before any remediation begins. Containment and evidence preservation only conflict when containment is done by pulling the plug.
2. The backups are 31 hours old and never tested. How does that change the containment decision, and what do you tell the plant director?
Read the worked answer
Untested backups make containment more valuable, not less. Your recovery point is already worse than assumed, which means you cannot afford to also lose the encrypted hosts - they may be the only intact copy of some data if the backup set is incomplete. Do not reimage or delete anything until images exist.
What to tell the director - quantities, not reassurance:
- If we do not isolate: worst case is all forty servers and a rebuild with no verified restore point.
- If we isolate now: three servers plus up to 31 hours of data, with the unknown being whether the 31-hour backup actually restores.
- We will begin a restore test of a single non-production host immediately, in parallel, and give a real recovery estimate rather than an optimistic one.
Offer a staged path: bring isolated segments back as they are individually verified, so the line resumes incrementally instead of waiting for a global all-clear. Do not commit to a restore time before the test completes - a missed promise costs more trust than the initial delay.
3. What are the decision points over the next 72 hours, and who owns each?
Read the worked answer
Name the owner, not the team. Unowned decisions default to whoever is loudest, which in this case is the cost argument.
| Decision | Owner | Why |
|---|---|---|
| Backup integrity and restore test | Infrastructure lead | Technical fact the whole plan depends on |
| Scope determination from EDR and flow telemetry | IR lead with the retainer responder | Needs tooling access, not business judgement |
| Whether to notify externally | Legal with an executive sponsor | The clock starts at reasonable belief, not at confirmed exfiltration |
| Customer and regulator communication content | Communications, approved by legal | Must not contradict the notification timeline |
| Line restart authorisation per segment | Plant director, on security sign-off | They own the cost, security owns the precondition |
Two failure modes to pre-empt. First, letting the retainer responder own business decisions - they own the investigation, not the 180,000-dollar-an-hour trade. Second, failing to log your own decisions with timestamps as you make them. The post-incident review will ask what you knew at 02:10 and when you knew it, and that trail cannot be reconstructed afterwards.