Nightly Archive Cleanup Deletes the Wrong Backups
A Debian 12 host runs /usr/local/sbin/prune-backups.sh from cron at 01:30. The script runs find /srv/backups -type f -name '*.tar.gz' -mtime +14 -delete, then emails the deletion count. Three jobs write into /srv/backups/app, /srv/backups/db, and /srv/backups/archive. Archive tarballs must be kept 90 days for audit. Last week an operator restored 38 audit tarballs into /srv/backups/archive/2024; they were older than 14 days. The next morning the script deleted all 38, exited 0, and reported 38 deletions. The archive directory is on the same ext4 filesystem as the job directories and is writable by the backup service account. A restore request is due in two days. The host has no snapshots, but the backup target keeps seven nightly rsync copies. You must change the script and recover the files without disrupting the three nightly jobs.
1. The script exited 0 and reported 38 deletions. Why did that not prove it deleted the right files, and what is your first recovery action?
Read the worked answer
The exit status only says find and the mail step completed without an error. -delete treats each removal as success, so a broad path plus a 14-day predicate is enough to produce a clean exit while deleting protected data. The count is also just the number removed, not a validation of scope.
First, stop the next cron run and any rsync job that could overwrite the retained copies: comment the cron entry or take the lock. Do not restore into the live archive yet. Identify the dated rsync copy from before the deletion, mount or copy it to a quarantine directory on a different filesystem, and compare its file list and checksums against the incident list. Restore only the missing 38 tarballs into a staging directory, verify with tar -tzf and sha256sum -c, preserve mtimes with cp -a, then move them into /srv/backups/archive/2024 after checking for name collisions. Keep the original directory listing and script output as evidence.
2. Rewrite the pruning logic so audit files are protected and concurrent writers cannot cause a bad deletion. Show the key controls.
Read the worked answer
Use an explicit allow-list, not the parent directory. For example:
#!/usr/bin/env bash
set -euo pipefail
umask 077
exec 9>/run/lock/prune-backups.lock
flock -n 9 || { echo 'another run active'; exit 0; }
mountpoint -q /srv/backups || { echo 'backup fs not mounted'; exit 1; }
for d in app db; do
find "/srv/backups/$d" -xdev -type f -name '*.tar.gz' -mtime +14 -print0 |
xargs -0 -r rm -f --
done
find /srv/backups/archive -xdev -type f -name '*.tar.gz' -mtime +90 -print
Key controls: the archive is never auto-deleted; the 90-day query only reports for audit approval. flock prevents overlapping cron or manual runs. mountpoint prevents deleting from an unmounted root directory. -xdev prevents crossing into unexpected mounts. Use -print0 and xargs -0 for spaces and newlines. Keep temporary backup names as .partial so they are not matched. Log every candidate and deletion to syslog, and make dry-run the default for new roots.
3. What test and monitoring would you put in place before this runs unattended again?
Read the worked answer
Build a disposable tree that mirrors app, db, archive, nested archive/2024, a filename with spaces, a partial file, and an old directory. Set mtimes with touch -d to exercise 13, 15, and 100 days. Run with a dry-run flag and assert the exact candidate list; then run against the copy and assert the archive is unchanged. Check shell syntax and static analysis (bash -n, shellcheck) and test the lock by starting two runs.
In production, log deletions with filename, size, and mtime to a central log, and alert if the count exceeds a baseline or if any path outside the allow-list appears. Monitor free space and the archive file count. Keep the rsync target append-only or on a different credential so a bad script cannot prune the recovery copies. Have a quarterly restore test, and require a second approval for any change to an archive root.