The Backup That Wasn't Tested
A regional retailer runs its order system on-premises with nightly full backups to a tape library and weekly copies to a cloud bucket. The DR plan says RTO 4 hours and RPO 24 hours. A ransomware incident encrypts the order database and the backup catalog server. The team discovers the cloud copies are also encrypted because the backup service account had write access to the bucket and the ransomware used the same credentials. The last verified restore test was 14 months ago. Tape rotation is offsite but the retrieval SLA is 48 hours. The business loses roughly $40k per hour of order-system downtime.
1. Assess the actual RTO/RPO against the stated plan and identify the failures.
Read the worked answer
The stated RTO 4h / RPO 24h is not achievable given the evidence:
- RPO failure: the cloud copies are encrypted, so the most recent usable copy is the offsite tape. If the last tape is up to a week old, actual RPO is days, not 24 hours.
- RTO failure: tape retrieval is 48 hours, already 12x the 4-hour RTO, before restore even begins. At $40k/hour, that is ~$1.9M in downtime before data is back.
- Root causes: (1) backup credentials had write access to the backup destination - a classic ransomware target; (2) no immutable/offline copy; (3) no recent restore test, so the plan was assumed, not verified. The plan described an intention, not a capability.
2. What backup architecture changes would make the RTO/RPO credible?
Read the worked answer
Apply 3-2-1-1-0 and least privilege:
- 3 copies, 2 media, 1 offsite, 1 offline/immutable, 0 errors verified.
- Immutable/object-lock cloud storage so the backup account cannot delete or encrypt retained copies; separate credentials for backup write vs. retention admin.
- Air-gapped or offline copy that ransomware cannot reach with production credentials.
- Faster restore path: keep a recent, restorable image locally or in cloud for the 4-hour RTO; tape is the last resort, not the primary.
- Credential separation: the backup service account should not be able to modify retention or delete; use a separate break-glass account for restore.
- Test cadence: quarterly full restore tests with measured time, documented results. Then re-baseline the RTO/RPO to what is actually demonstrated, and tell the business the real numbers.
3. How do you handle the immediate incident and the credibility problem with leadership?
Read the worked answer
Immediate: isolate affected systems, preserve evidence, engage IR, and begin tape retrieval now (48h clock). Do not pay or negotiate before scoping. Determine whether the backup catalog can be rebuilt from tape. Credibility: present the facts plainly - the plan's RTO/RPO were never validated, and the backup copies were reachable by the attacker. Own the gap, give the real recovery timeline, and commit to a dated remediation plan with owners. Avoid blaming individuals; the failure is architectural (shared credentials, no immutability, no testing). Governance: add restore testing as a recurring control with evidence, and require DR plan sign-off only after a successful test. Report the financial exposure ($40k/hr) so leadership understands why the investment is justified.