Recovering a Dropped Collection When the PITI Window Has Expired
A 2.1 TB Atlas M30 replica set uses Cloud Backup: continuous backups with the point-in-time window configured to 24 hours, plus snapshots every six hours on the hour, retained seven days. On Saturday at 02:40 a release script ran an unfiltered deleteMany({}) against shop.line_items (about 180 GB, 60 million documents) and exited successfully. Nobody noticed. On Monday at 08:20 the support team reports that order history is empty. Writes to other collections - orders, customers, payments - have continued normally throughout, and the application has been inserting new line_items documents since Sunday evening as part of a partial backfill. The RTO target for this system is four hours. The team wants to run the point-in-time restore to 'just before 02:40'.
1. Evaluate the proposed point-in-time restore. What RPO can you actually achieve, and which restore would you choose?
Read the worked answer
The proposal does not work. Point-in-time restore can only reach back inside the configured window, and the window here is 24 hours measured from now (Monday 08:20). Saturday 02:40 is roughly 54 hours ago, so that point is not selectable. Even if it were, an in-place PITI restore would roll the entire deployment back 54 hours, destroying every order, payment, and customer write since - a far bigger incident than the one being fixed.
What is available: snapshots every six hours, retained seven days. The newest snapshot before the deletion is Saturday 00:00, which is 2 hours 40 minutes before the delete. So the best achievable RPO for the damaged collection is ~2h40m: line_items changes made between 00:00 and 02:40 on Saturday are gone unless they can be reconstructed from an application-side record.
Choice: restore the Saturday 00:00 snapshot to a new cluster, then extract only shop.line_items and merge it into production. This preserves the 54 hours of legitimate writes to other collections and keeps production serving traffic while the 2.1 TB restore runs in the background - the only path that fits a four-hour RTO.
Also do first: take an on-demand snapshot of the current production state for forensics, and disable the account the release script used.
2. Write the execution plan for the restore and the data merge, including how you avoid overwriting post-deletion data and how you verify success.
Read the worked answer
- Contain: disable or rotate the release script's credentials; confirm nothing else is still deleting.
- Forensics snapshot: on-demand snapshot of production now, so the recovery itself is reversible.
- Restore to a new cluster: same major version, same region, tier large enough to hold 2.1 TB. Name it clearly (e.g.
restore-sbx). Expect a multi-hour restore; start it before anything else. - Extract only the damaged namespace:
mongodump --uri <restore-sbx> --nsInclude 'shop.line_items' --gzip --archive=line_items.gz. Do not restore the whole snapshot over production. - Land it in a staging namespace, not the live one:
mongorestore --nsFrom 'shop.line_items' --nsTo 'shop.line_items_restore'. Documents created by the weekend backfill now collide with restored_ids, and staging lets you inspect that before it becomes an outage. - Merge with conflict rules: iterate over tenants (batches of 100k documents) and run
$mergeintoshop.line_itemswithon: ["_id"],whenMatched: "keepExisting",whenNotMatched: "insert". keepExisting means a document modified after the deletion is not clobbered by the stale snapshot copy. Read from the restore cluster, write to production during a low-traffic window. - Verify: document counts per tenant against the Saturday 00:00 baseline, spot-check known orders, confirm the app can load a 30-day history for three pilot accounts, and re-check index usage. Drop the staging namespace and, after a week, the restore cluster.
3. What changes would you make so the next incident is caught in minutes and capped in blast radius?
Read the worked answer
Detection gap is the real failure. Fifty-four hours passed because alerting watched cluster health, not data. Add a lightweight synthetic check - a scheduled job that counts documents per tenant over the last hour and alerts on a drop to zero or a deviation beyond normal variance. Native Atlas alerts on CPU, connections, or disk would never have fired for a logical delete.
Blast radius
- The release script used a broad-privilege database user. Give it a user scoped to its own database with only the privileges it needs, so it cannot run unrestricted deletes on production collections.
- Enforce in code review and in the data-access layer that bulk deletes require a mandatory filter; reject empty-filter deletes at the library level.
- Enable an Atlas backup compliance policy so snapshots cannot be deleted or shortened by any admin. Note it is one-way and not retroactive - decide before the incident, not during it.
Recovery posture
- Extend the PITI window from 24 hours to the maximum retention you can justify; the window is not retroactive, so this buys nothing for Saturday's deletion but everything for the next one.
- Run a quarterly restore drill and record measured RTO for the actual data size; the assumed four hours has never been tested at 2.1 TB.
- Ship audit logs to S3 so 'who ran what and when' is answered from evidence rather than guesswork.