Encrypted Backup Catalog, a Nine-Day-Old Tape, and an Eight-Hour RTO
Coastal Foods operates three plants on a single-site VMware ERP. The stated RTO is eight hours and the RPO for financials is one hour. Backups are a nightly full to a NAS plus a weekly tape sent to an offsite vault, with a four-week tape rotation. The NAS is domain-joined and its administrator credentials are shared with the backup service account. On a Tuesday at 03:00 a ransomware note appears on 60 servers. The team finds the NAS backup catalog encrypted, and the most recent tape that restores cleanly is nine days old. The DR site has hardware, but the failover runbook has never been validated and the last test, 14 months ago, failed at the database tier. The cyber insurer requires notification within 72 hours of confirmed incident. Every day of production stoppage costs about $400,000 across the three plants.
1. What are the achievable RTO and RPO here, and how do you sequence recovery?
Read the worked answer
Start by stating the honest gap: the design promised an 8-hour RTO and a 1-hour RPO; the actual position is a 9-day RPO for anything that only exists on tape. Do not brief the business on the design figures - brief them on the real ones and revise the timeline as facts land.
Sequence:
- Preserve before wiping. Keep the encrypted NAS and the affected VMs offline but intact. Some ransomware leaves recoverable shadow copies, keys, or a retrievable catalog, and the forensic image is needed for the insurance claim and any law-enforcement report.
- Establish what is actually clean. Identify data that never depended on the NAS: cloud email and collaboration, the ERP database's own dumps if they exist on separate storage, plant-level local systems. This frequently shortens the outage more than any heroic tape restoration.
- Restore in dependency order into an isolated clean VLAN: identity and DNS first, then the ERP database, then application tier, then file shares, then plant MES interfaces. Restore, scan, verify, then attach to production - never restore an infected VM directly.
- Declare the disaster formally and give the business a realistic estimate, probably 2–4 days for full production rather than 8 hours. A half-restored ERP that corrupts financials is worse than a clear recovery plan.
- Notify the insurer within 72 hours with facts known at the time; the legal and regulatory clocks may be separate and possibly shorter, so loop in counsel on day one.
The judgement point is resisting the pressure to promise 8 hours because it is written down.
2. Which specific backup architecture decisions turned a server incident into a company-wide outage?
Read the worked answer
Four design choices, each individually defensible and collectively fatal.
1. Shared credentials and domain membership. The NAS was domain-joined and its admin password was reused for the backup service account. One credential compromise gave the attacker the production estate and the backup target. Backups must live in a separate identity and trust plane: dedicated local accounts or a separate directory, MFA on the backup console, no shared secrets, and the backup service account restricted to the backup target only.
2. No offline or immutable copy. Every night's backup was reachable from production. A ransomware operator with a week of dwell time encrypts the source and the copy. The fix is object-lock or an immutable appliance, plus a genuinely offline copy, following 3-2-1-1-0: three copies, two media, one offsite, one offline or immutable, zero verified errors.
3. A four-week rotation with no verification. Nine days was the newest clean tape, so an infection older than the rotation window would have been in every tape in the vault. Retention must exceed any plausible dwell time, and restores must be tested - a backup that has never been restored is a hypothesis, not a backup.
4. Single site with an untested DR capability. The DR site existed but the runbook had never been validated and the database tier failed the last test and was never fixed. Hardware without a rehearsed procedure is not recoverability.
Also introduce pre-backup anomaly detection (volume change, mass deletion, encryption indicators) so backup jobs fail loudly rather than overwriting the last good copy.
3. The insurer and the board both want assurance. What testing programme and evidence do you put in front of them?
Read the worked answer
Both audiences are asking the same question - "will you recover next time, and can you prove it?" - so the answer is a test programme with records, not a plan document.
Fix the known failure first. The database tier is the reason the last test failed; remediating it and re-testing is the first deliverable, and it should precede any new tooling.
Test design:
- Define success criteria per tier: time to restore, data loss measured in hours, and business verification by an actual process owner, not just "the service started."
- Test from the DR side using the written runbook, with the runbook's author excluded from driving it - if a competent engineer cannot follow it unaided, it is not a runbook.
- Include the ugly cases: restore from tape into a clean environment, identity services unavailable, and the backup console compromised. These are exactly the ones that broke this time.
- Record actual restore durations and use them to revise RTO/RPO upward to something defensible, then get the business to accept the revised figures.
Evidence for the insurer: immutable or offline backup configuration and a screenshot of object-lock retention, MFA enforcement on the backup platform, segmentation and EDR coverage, the incident timeline, the last test report with measured durations, and a dated remediation plan with owners.
Evidence for the board: one page - current achievable RTO/RPO versus stated, cost per day of outage, and the two or three funded actions that close the gap. Be explicit that the previous figures were aspiration, not capability, and that the 8-hour claim will be withdrawn until a test demonstrates it.