A Backup Dashboard That Is 100% Green and Cannot Meet RTO
A 2 TB transactional database is backed up with a nightly incremental at 02:00 that takes 25 minutes and a weekly full on Sunday at 01:00 that takes 5 hours. Backups go to an offsite object store and are retained for 30 days. The backup dashboard has reported 100% job success for 90 consecutive days. The business states an RTO of 4 hours and an RPO of 15 minutes. The last restore test was 14 months ago. Last week's test restore, run by hand, took 6 hours 20 minutes end to end: 40 minutes to provision, 3 hours 53 minutes to transfer 2 TB at an observed 150 MB/s, and 1 hour 50 minutes to apply and recover. During validation, 3 of the 30 available restore points could not be read.
1. Which stated objectives are actually unachievable, and by how much? Show the numbers.
Read the worked answer
RPO: unachievable by roughly two orders of magnitude. The only backup taken is the 02:00 incremental. A failure at 01:59 loses 23 h 59 min of committed transactions against a 15-minute target - about 96 times the tolerance. The RPO is not "at risk"; it is not addressed by the current design at all.
RTO: unachievable, and structurally so.
- Observed restore: 0:40 + 3:53 + 1:50 = 6 h 23 min against a 4-hour target, an overrun of 2 h 23 min.
- Worse, the overrun is not a tuning problem. Fix the throughput to infinity and you still have 2 h 30 min of provisioning plus apply time, leaving only 1 h 30 min to move 2 TB.
- Required transfer rate to hit 4 hours: 2,097,152 MB ÷ 5,400 s ≈ 388 MB/s, about 2.6× what the offsite link delivers today.
- So meeting RTO by making the transfer faster is not available on this architecture. It needs a different recovery technique, not a bigger pipe.
Effective current posture: RPO ≈ 24 hours, RTO ≈ 6.5 hours, plus a caveat that 10% of restore points do not read. Two of the three numbers the business is relying on are fiction.
2. Why is a 100% green dashboard not evidence of recoverability, and what should replace it?
Read the worked answer
Why it is not evidence
The dashboard measures whether the backup job exited successfully. That is a statement about the backup process, not about the artifact or the restore path. Everything that actually determines recoverability sits downstream of the job exit code:
- whether the object store can serve 2 TB at a usable rate against a 4-hour clock;
- whether the restore procedure takes 2 h 30 min of fixed overhead before data moves;
- whether the restore points decrypt, decompress, and apply cleanly - and 3 of 30 do not;
- whether anyone still knows the procedure. The last test was 14 months ago.
A 14-month-old restore test also means the 6 h 23 min figure is a single unvalidated observation, likely optimistic: the person running it knew the system and did it by hand.
Replace with
- Restore test pass rate, with restore points randomly selected from the retention window rather than the most recent.
- Measured time-to-restore, end to end, reported against RTO and trending.
- Percentage of restore points verified readable, so 3 failures in 30 reads as 90%, not as a footnote.
- A per-quarter full restore rehearsal performed by whoever is actually on call, not by the person who built the system.
Keep job success - as a leading indicator, not as the compliance number.
3. What is the cheapest change that closes the RPO gap, and what does it do to RTO?
Read the worked answer
Cheapest RPO fix: continuous transaction-log shipping, archived to the offsite store every 5 minutes. This is a configuration change on the existing database, not a new product, and it addresses the actual defect - the only thing between backups is 24 hours of nothing. Archiving every 5 minutes brings worst-case loss to about 5 minutes, inside the 15-minute target with margin, and it also makes point-in-time recovery possible for the corruption case that logical backups handle poorly.
The trap: it makes RTO worse.
Restoring to a point in time means restoring a base backup and then replaying log. That moves work into the apply step, which is already 1 h 50 min, and it adds a dependency on log continuity - one missing archive segment and you cannot reach the target point. So the change must be paired with:
- Log archive monitoring that alerts on the absence of a segment, not only on job failure.
- A restore path that does not begin with a 3 h 53 min offsite transfer - a periodically refreshed local base image or block-level restore from a closer copy.
- A re-measured end-to-end restore time, because the 4-hour RTO is still unmet and tightening RPO will not fix it.
Sequence: fix RPO first because it is cheap and removes the largest exposure; treat RTO as a separate design problem requiring a faster recovery path, and tell the business now that 4 hours is not achievable with the current architecture rather than at the next test.