Cross-Region Backup and RPO for OSS-Backed Application Data
A SaaS provider stores user-generated files in an OSS bucket in the China (Hangzhou) region. The bucket has versioning enabled and a lifecycle rule that transitions objects to Infrequent Access after 30 days and deletes noncurrent versions after 90 days. The business requires an RPO of 15 minutes and an RTO of 2 hours for a regional outage. Daily backups are taken by a script that copies objects to a second bucket in the same region. Last month, an operator accidentally deleted a prefix containing 40,000 objects; recovery took 6 hours because the script had to re-upload from a local cache.
1. Evaluate whether the current design meets the stated RPO/RTO and identify the gaps.
Read the worked answer
It does not meet the requirements.
- Same-region backup: both buckets are in Hangzhou, so a regional outage takes out primary and backup. RTO of 2 hours for a regional disaster is not achievable.
- Daily cadence: RPO is effectively 24 hours, not 15 minutes. Any object created after the last run is unprotected.
- Versioning helps accidental deletion but the 90-day noncurrent-version expiry and the fact that a prefix delete creates delete markers means recovery depends on restoring versions; the 6-hour recovery shows the process was manual and cache-dependent, not a tested restore path.
- Lifecycle to IA does not affect durability but does affect retrieval latency/cost during a bulk restore.
Gaps: no cross-region replication, no continuous/incremental protection, no documented and rehearsed restore procedure, and no monitoring of replication lag.
2. Design a revised protection scheme that meets RPO 15 minutes and RTO 2 hours, including how you would validate it.
Read the worked answer
Proposed design:
- Cross-region replication from the Hangzhou bucket to a bucket in a second region (e.g., Zhangjiakou or Shenzhen). Enable versioning on the destination as well. Replication is asynchronous; measure and alert on replication lag to confirm it stays within the 15-minute RPO. Use replication rules scoped to the prefixes that matter and exclude transient/temp prefixes to control cost.
- Retain versions longer on the destination (e.g., 180 days) and add an MFA-delete or retention policy so a compromised or mistaken operator cannot purge recovery copies.
- Inventory and restore tooling: maintain an object inventory and a documented restore runbook. For bulk restore, use OSS batch processing or a parallel copy job rather than a local cache. Pre-stage a restore environment so RTO is dominated by data transfer, not setup.
- RTO math: estimate restore throughput for the largest expected dataset; if 40,000 objects at average size X cannot be restored in under 2 hours, add a warm standby or a secondary copy closer to the recovery site.
Validation:
- Run a quarterly game day: delete a test prefix in the primary, restore from the replica, and time it end-to-end.
- Inject a simulated regional failure (block primary access) and confirm the application can point to the replica region.
- Alert on replication lag, failed replication operations, and bucket policy changes.