The Backup That Only Works When You Run It

There are at least four independent defects, and the claim that it exits 0 needs checking first: with set -euo pipefail, aws: command not found should produce a non-zero exit unless the failing command is in a pipeline whose status is masked, or the exit status being observed is cron's rather than the script's. Confirm by adding echo exit=$? at the end and by checking the journal or /var/log/syslog for the CRON entry.

Likely causes and checks:

• PATH: cron uses /usr/bin:/bin, so aws in /usr/local/bin is not found. Confirm with env -i HOME=/home/backup PATH=/usr/bin:/bin SHELL=/bin/sh /home/backup/backup.sh and with command -v aws under that environment.
• Working directory: cron starts in the user's home, but the script uses $(pwd), and the target directory may not exist. Confirm by logging pwd and running with cd / to reproduce.
• Credentials: the AWS profile may rely on an interactive login or a file under a HOME that differs under cron. Confirm with aws sts get-caller-identity under the cron environment.
• Locking and filename collisions: two runs wrote the same filename, so one truncated or overwrote the other. Check timestamps, the .part file, and whether both runs started within the same minute.

Reproduce with env -i before changing anything; that single step separates environment faults from logic faults.

Crontab: use absolute paths, set the environment explicitly, and log with a timestamp. Define PATH=/usr/local/bin:/usr/bin:/bin, HOME=/home/backup and SHELL=/bin/bash at the top of the crontab, then invoke /home/backup/backup.sh with output to a dated log. Remember that % in a crontab is special and must be escaped if used.

Script properties:

• Keep set -euo pipefail and an explicit IFS, and add a trap that logs the failing line and removes partial files on exit.
• cd to an explicit directory or use absolute paths everywhere; mkdir -p the dump directory; do not derive paths from pwd.
• Single-instance locking: exec 9>/var/lock/backup.lock; flock -n 9 || exit 1, so the manual run and the cron run cannot collide.
• Pre-flight checks: free space at least twice the expected dump size, database reachable, credentials valid via aws sts get-caller-identity.
• Verification before upload: check the dump is non-zero and passes gzip -t or tar -tzf; compute a checksum and compare it after upload.
• Failure detection: exit non-zero on any failure and send an alert; a backup that silently fails is worse than one that does not run.
• Retention: delete local dumps only after a successful upload and checksum check, and apply a documented retention policy in object storage.

Verification should reproduce cron's environment, not your interactive shell. Run sudo -u backup env -i HOME=/home/backup PATH=/usr/local/bin:/usr/bin:/bin SHELL=/bin/bash /home/backup/backup.sh and check the log, exit status, dump size, checksum and object-storage listing. Then trigger a real cron run by setting the schedule a few minutes ahead, and confirm the lock prevents a second concurrent run by starting a manual run at the same time. Finally, restore the dump to a scratch database and compare row counts or checksums; an unverified backup is only a hypothesis.

Add a dead-man's-switch alert that fires if no successful backup marker has been written for 26 hours, plus alerts on job exit status and artifact age in object storage.

Documentation: purpose and RPO/RTO; schedule and retention; the account and credential source; exact commands to run, verify and restore; expected runtime and size; dependencies such as the aws CLI path and database access; known failure modes including PATH, locking and disk space; where logs live; and who owns the job. Include the restore-test date and result, and note that the previous undocumented state caused this incident. Store it with the other runbooks, not only in a ticket.