Clean-Room Restore Sequencing Under a 12-Hour RTO
A logistics company's VMware software-defined data center has 42 hosts and 1,100 VMs. Ransomware encrypted 70% of VMs, including vCenter. Backups are immutable for 14 days and stored in an isolated recovery environment (IRE) with 6 hosts and 1.2 PB. The last clean backup is 3 days old. Dependency mapping shows 28 applications; tier-0 includes DNS, AD, load balancers, and a mainframe gateway. RTO is 12 hours for tier-0 and 48 hours for tier-1. The IRE has 30% spare capacity. The team wants to restore all VMs simultaneously to meet RTO. There are 2,400 restore tasks queued; the IRE can run 40 concurrent restores. Network egress from IRE to production is 5 Gbps. A test restore of 100 VMs took 4 hours. The security team requires malware scanning and integrity validation before any restored VM connects to production.
1. Build a restore sequence for tier-0 and justify each ordering decision.
Read the worked answer
Restore in dependency order, not alphabetical order: DNS first, then AD domain controllers, then load balancers, then the mainframe gateway, then the applications that depend on them. DNS comes first because AD and almost every application resolve names; restoring AD without DNS creates authentication failures that look like corruption. AD must come from a clean backup and be validated before load balancers route traffic to it. Load balancers come next so that tier-0 services can be reached by stable VIPs, but they should be configured in a maintenance or drain state until health checks pass. The mainframe gateway follows because it often carries a fixed session limit and must be started before dependent order-processing apps. Only then restore tier-0 application VMs in waves.
The 12-hour RTO applies to tier-0, not all 1,100 VMs. Restoring everything simultaneously would saturate the 40 concurrent restore slots and the 5 Gbps egress, delaying tier-0 behind non-critical workloads. The sequence should explicitly defer tier-1 and tier-2 restores until tier-0 health checks pass.
2. How should the team validate clean state and prevent re-infection before connecting restored VMs to production?
Read the worked answer
Use a gated promotion process. First, restore into the IRE with network isolation and no production routing. Second, run malware scanning and integrity validation against the backup metadata and the restored file system; compare known-good hashes for critical binaries and check for suspicious services, scheduled tasks, and startup items. Third, verify application consistency, such as database transaction log replay and service health, without allowing external network access. Fourth, apply security patches and rotate credentials before promotion. Fifth, connect to production through a controlled staging VLAN and monitor for anomalous outbound traffic.
The key judgment is that a clean backup is necessary but not sufficient; the restore process itself can reintroduce malware from a compromised backup or from an infected IRE host. The IRE hosts should be rebuilt or scanned between waves, and promotion should require a second approver. If any tier-0 VM fails validation, stop the wave and re-check the backup selection rather than proceeding to meet the clock.
3. Identify the bottleneck for the 12-hour tier-0 RTO and propose a mitigation that does not compromise validation.
Read the worked answer
The arithmetic points to restore concurrency. A test of 100 VMs in 4 hours equals 25 VMs per hour. If tier-0 is, for example, 200 VMs, that is 8 hours of restore time before validation, leaving only 4 hours for scanning, patching, and promotion within the 12-hour RTO. The 40 concurrent restore limit is the bottleneck, not the 5 Gbps egress, because 25 VMs per hour at an average of, say, 100 GB per VM is about 2.5 TB/hour, which is well below 5 Gbps (about 2.25 TB/hour at 5 Gbps, so egress may also bind; measure actual VM sizes).
Mitigation: pre-stage tier-0 backup data in the IRE, use instant recovery or mount-based restore for validation, and increase IRE restore concurrency only after measuring target and network headroom. Do not skip scanning. Run validation in parallel on separate IRE hosts and promote in small batches. If the measured egress is the real limit, restore tier-0 locally in the IRE and run production from there temporarily, then migrate back during the tier-1 window.