An Eleven-Hour Dump, a Dropped Collection, and a Four-Hour RTO
An e-commerce platform runs a sharded cluster: three shards, each a three-member replica set, plus a three-member config server replica set. Total data is 4.2 TB and the oplog window is five hours at peak. Backups are a nightly mongodump launched at 02:00 from a mongos with no special options; it takes 11 hours and pushes replica-set CPU to 85 percent, degrading customer traffic until early afternoon. At 09:14 an engineer ran drop() on the orders collection by mistake; the error was noticed at 13:40. The stated targets are RPO 15 minutes and RTO 4 hours. There are no filesystem snapshots today, and the team has never performed a restore test.
1. Can the team recover the dropped orders collection from last night's dump? Walk through what is and is not possible.
Read the worked answer
Partly, and only with care, because that dump is not a point-in-time backup. A mongodump without --oplog walking a 4.2 TB sharded cluster over 11 hours captures different collections, and different chunks of the same collection, at wildly different times. Some of the orders data was dumped before 09:14 and some after the drop, so a naive restore gives a torn collection with an arbitrary partial dataset and no indication of where the seam is.
What is possible: restore into a scratch cluster, never in place, and treat it as forensic recovery. Identify which per-shard dump directories contain the orders collection, restore those into a staging sharded cluster with matching config metadata so the collection is recreated with its chunk layout, then reconcile. Expect to lose everything written between the snapshot of each shard and the drop time - for the earliest-dumped shards that could be most of the day.
What is not possible: replaying the oplog to a precise time. The dump carries no captured oplog, and mongorestore --oplogReplay is a per-replica-set mechanism that brings one set to a consistent point; there is no supported cluster-wide point-in-time replay through mongodump/mongorestore, because each shard's timeline and the config server metadata would have to be reconstructed together. Even though the 09:14 event sits inside the five-hour oplog window, there is nothing to replay against.
The honest answer to give the business: we can recover an approximate orders collection with a known and possibly large gap, and we cannot prove what is missing. That is the cost of not having had a real backup.
2. Design a backup architecture that meets RPO 15 minutes and RTO 4 hours for this 4.2 TB cluster.
Read the worked answer
RPO 15 minutes rules out nightly logical dumps by three orders of magnitude. You need either continuous oplog-based backup or frequent block-level snapshots.
Preferred: a backup system that tails the oplog continuously - Ops Manager, Cloud Manager, or Atlas if migration is on the table. It gives restore-to-a-second, an inventory of what exists, coordinated cluster-wide consistency, and an automated restore workflow that fits a 4-hour RTO. This is the only option that genuinely gives a 15-minute RPO with a defensible story for a sharded cluster.
Alternative if the tooling is unavailable: filesystem snapshots of each data-bearing node, taken every 15 minutes from secondaries, at the volume level (LVM, EBS, or your storage equivalent) with the journal files included. A snapshot that includes the journal is crash-consistent and recoverable; one without the journal may not be. Take snapshots on all three shards and the config servers within the same window, and either stop the balancer during the window or accept that a chunk may have been migrating and reconcile at restore. Keep a rolling set of snapshots - 96 per day at 15 minutes plus daily retention - and pre-provision the IOPS to restore in parallel across shards.
Regardless of mechanism:
- Restore from secondaries, never from the primary, and throttle the backup I/O so it does not repeat today's 85 percent CPU problem.
- Verify restores by actually restoring, not by checking exit codes.
- Keep backups in a separate failure domain from the cluster, and confirm the EU data residency rules cover the backup location too.
3. Write the restore test and runbook you would put in place, including the traps specific to sharded clusters.
Read the worked answer
Run the test quarterly against a production-sized copy, in an isolated network, and time it end to end. The deliverable is a measured RTO per data size, not a document saying 'restore from backup'.
Runbook:
- Declare the restore and freeze writes or accept a known point to stop at. Record the target time.
- Restore the config server replica set first, from its own consistent point. Config metadata defines which shard owns which chunk range; restoring shards without it, or with metadata from a different time, produces a cluster that starts but returns wrong or missing data.
- Restore each shard's replica set from its backup, with the balancer stopped (sh.stopBalancer()) so nothing moves mid-restore.
- Verify replica sets are healthy and in sync, then start mongos routers, then re-enable the balancer, then re-enable writes.
- Validate: collection counts against the backup manifest, a checksum or sampling query per critical collection, and application smoke tests. Do not declare success on 'mongod started'.
Traps to write into the runbook:
- Restoring a sharded cluster into a differently sharded topology is not supported; the shard count and shard key must match.
- A restore of a subset (one collection) still needs matching metadata, which is why the single-collection recovery in production is so awkward.
- Snapshot restores must include the journal, and secondaries must be restored consistently with the primary they will sync from or they will roll back or resync.
- Time the restore with realistic parallelism: seeding three shards simultaneously needs the disk throughput, and a 4-hour RTO on 4.2 TB requires roughly 300 MB/s sustained plus index rebuild time.
- Test the human path too: who has credentials, who can approve, and who declares the RPO breach.