Backup Window Overrun and Tier-1 Snapshot Timeouts
A payments processor protects 240 TB across 1,800 VMs. Tier-1 ledger VMs require a 15-minute RPO, 4-hour RTO, and 90-day immutable retention. Tier-2 analytics VMs require 24-hour RPO, 24-hour RTO, and 30-day retention. The current policy runs a weekly full and 4-hour incrementals over a 10 Gbps backup network; measured dedupe is 3.5:1 and the backup window is 6 hours. After adding 30 TB of container images, the window overruns by 55 minutes and 11 tier-1 VMs report 'snapshot consistency group timeout' during peak I/O. The team proposes moving tier-2 to 12-hour incrementals and doubling concurrent streams from 8 to 16. The target can absorb 4 TB/hour sustained; the network achieves 9 Gbps in practice. A ransomware tabletop requires tier-1 backups to remain immutable for 90 days and any change must not extend the 6-hour window.
1. Evaluate whether the proposed change meets the tier-1 RPO and the 6-hour window. Show your reasoning and identify the most likely cause of the consistency timeouts.
Read the worked answer
The proposal does not fix the tier-1 problem. Doubling concurrent streams from 8 to 16 increases simultaneous I/O against the same datastores and backup target, which makes consistency-group timeouts more likely, not less. The 30 TB addition at 3.5:1 dedupe adds roughly 8.6 TB of physical backup data. At the measured 4 TB/hour target ingest rate, that is about 2.15 hours of capacity, so raw capacity is not the binding constraint; the 55-minute overrun is a scheduling and contention problem.
Reasoning steps:
- Current window is 6 hours; overrun of 55 minutes means actual elapsed time is about 6h55m.
- Tier-1 RPO is 15 minutes; the 4-hour incremental schedule already misses that unless tier-1 uses a separate, more frequent policy. The proposal leaves tier-1 at 4 hours, so it fails the RPO independent of window length.
- Moving tier-2 to 12-hour incrementals reduces tier-2 job frequency but does not reduce peak concurrency for tier-1 VMs.
Recommendation: split tier-1 and tier-2 into separate SLAs and streams; stagger snapshot start times across datastores; move container images to a 24-hour RPO SLA; pilot stream increases on one cluster before global change.
2. What retention and immutability changes would you make to satisfy the 90-day tier-1 requirement, and what trade-offs must be accepted?
Read the worked answer
Set a dedicated tier-1 SLA with 90-day immutable retention and a compliance lock or legal hold that prevents deletion or shortening of the chain. Ensure the incremental chain does not expire before 90 days; if the platform expires incremental metadata before the immutability period, restore points become unusable even if the data blocks remain. Validate that immutability applies to the backup metadata and the underlying objects, not just the job record.
Trade-off analysis:
- Storage cost rises because tier-1 change data must be kept for 90 days regardless of aging. Estimate using daily change rate multiplied by 90, divided by dedupe, and compare with current 30-day retention.
- Early deletion is blocked, so operational mistakes cannot be cleaned up quickly; use a break-glass process with dual approval and audit.
- Immutability can conflict with storage capacity alerts; monitor headroom against the 4 TB/hour ingest ceiling and the 6-hour window.
The tabletop requirement is not met by simply setting a 90-day label; the platform must enforce it and prove it with an audit report.