Backup window and restore chain for a 40 TB database
A financial reporting platform runs a 40 TB database on a high-end hybrid array. The backup policy is weekly full on Sunday and daily incremental, with a 30-day retention chain. Backups run over a dedicated 10 GbE network to a deduplicating appliance; measured throughput is 500 MB/s after deduplication and compression. The weekly full takes 22 hours, exceeding the 4-hour backup window, so it currently overlaps Monday production. Daily incremental changes average 6 TB and finish in 3.5 hours. A recent restore of a 2 TB tablespace took 9 hours because the appliance had to rehydrate data from a 12-deep incremental chain. The business RTO is 4 hours and RPO is 1 hour. The database team wants to keep 30 days of recovery points but cannot extend the window. Network peak is 5.5 Gbps and the source array is at 65% utilization.
1. Calculate the effective throughput required to complete a 40 TB full backup in 4 hours, and compare it with the current 500 MB/s. What does this imply about simply adding network bandwidth?
Read the worked answer
Required throughput for 40 TB in 4 h: 40 TB / 4 h = 10 TB/h = 2.78 GB/s = 22.2 Gbps. Current 500 MB/s = 0.5 GB/s = 4 Gbps; 40,000 GB / 0.5 GB/s = 80,000 s ≈ 22.2 h. Even a single 10 GbE at full line rate (~1.25 GB/s) would need 32,000 s ≈ 8.9 h for 40 TB. So the 4-hour window cannot be met by streaming a full backup over one 10 GbE, and the source array at 65% utilization plus the dedup appliance also constrain throughput. Adding bandwidth might reduce the 22-hour full, but it does not fix the restore-chain problem or the appliance rehydration rate. The design must reduce the amount of data moved in the window, not just move it faster.
2. Propose a data protection redesign that meets RTO 4 hours and RPO 1 hour while retaining 30 days. Address the restore chain.
Read the worked answer
Use hourly storage snapshots for RPO 1 h, retain 24–48 h locally, and replicate snapshots to the secondary site for disaster recovery. Keep daily incremental backups to the dedup appliance, but replace the 30-deep incremental chain with weekly synthetic fulls (or forever-incremental with periodic synthetic fulls) to cap chain depth at 7–14. For 30-day retention, keep 7–14 days on the appliance and archive monthly fulls offsite. For restore, mount a snapshot or instant clone of the 2 TB tablespace and recover on the clone; 2 TB in 4 h needs 139 MB/s sustained, whereas the observed 9 h restore delivered only about 62 MB/s. Validate that snapshot/clone recovery meets the 4-hour RTO and that the clone can be resynced or discarded. Do not rely on hourly snapshots as the only copy; keep the appliance copy for corruption and site loss.
3. What risks does this design add, and how would you test it before relying on it?
Read the worked answer
Risks: snapshot space growth and pool exhaustion; snapshot overhead on the production array; clone capacity and host mapping errors; appliance dedup rehydration still slow if the chain is long; and retention gaps if snapshot and backup schedules are misaligned. Also, a snapshot is not independent of the source pool, so a pool loss can take both source and snapshot. Test by performing a timed restore of the 2 TB tablespace from a snapshot/clone and from the appliance, measuring end-to-end RTO including database recovery. Run a controlled snapshot deletion and pool-high-water test, verify replication to the secondary site, and confirm 30-day recovery points with a sample restore. Monitor snapshot consumption daily and alert at 70% and 85% of the snapshot reserve.