The Backup That Was Never Restored
An engineering team stores a production database's nightly snapshots in object storage in the same region and account as the live database. Snapshots have run successfully for 14 months. During an incident, an operator accidentally deletes the production database and its storage volume. The team attempts to restore from the most recent snapshot and discovers that the snapshot process captured the volume but the application's configuration and encryption keys were stored separately and were also deleted. The last usable full restore is 11 months old because incremental snapshots depend on a base image that was rotated out. Compliance requires a recovery point objective of 24 hours and a recovery time objective of 4 hours. Nobody has tested a restore since initial setup.
1. Identify the distinct failure modes in this backup design and rank them by severity.
Read the worked answer
Four distinct failure modes, ranked:
- Untested restore path (most severe). The team believed backups worked because jobs succeeded. Job success measures write completion, not recoverability. This is the failure that turned a recoverable incident into an 11-month data loss.
- Incomplete backup scope. The snapshot captured the volume but not configuration or encryption keys. A backup is only as complete as its dependency graph; keys and config are part of the recoverable unit.
- Dependency on a rotated base image. Incremental chains that rely on a base that gets deleted are silently fragile. Retention policy for the base must outlive the retention of every increment that needs it.
- Same-region, same-account placement. This is a blast-radius problem - a single account compromise or regional event takes out both source and backup. Severe, but it did not cause this incident.
The ranking matters: fixing #4 first is a common instinct, but #1 and #2 are why the restore failed here.
2. Given a 24h RPO and 4h RTO, what changes would you require before signing off on the design?
Read the worked answer
Sign-off requires evidence, not intent. Required changes:
- Restore rehearsal on a schedule. A restore into an isolated environment at least monthly, timed and documented. RTO is a measured number; 4 hours is unproven until someone has done it under a clock.
- Backup the full dependency set. Configuration, secrets/keys (via a managed key service with its own backup and access controls), and infrastructure definitions must be in the recoverable unit. Store keys so that deleting the database does not delete the ability to decrypt its backup.
- Fix the incremental chain. Either move to a snapshot model without a fragile shared base, or enforce retention so the base outlives every dependent increment. Verify by attempting a restore from the oldest retained increment.
- Cross-region or cross-account copy. To meet 24h RPO against regional loss, replicate backups out of the primary region/account. Same-account deletion protection (object lock, versioning, MFA-delete) also reduces the accidental-deletion class.
- Alarm on backup age and restore-test failure, not just job exit code.
Only after a timed restore passes should the design be considered compliant.