A Backup Script That Behaves Differently Under Cron

cron does not run a login shell. The job starts with a minimal environment - on RHEL, PATH is normally just /usr/bin:/bin - plus root's crontab PATH= line if one was set. /usr/local/bin is not in that default PATH, so notify-admin is simply 'command not found'. Because the script's last line is echo backup finished, the script's exit status is 0 and cron has no reason to mail anything.

Prove it without guessing:

crontab -l                        # is there a PATH= line?
crontab -l | grep -i mailto
# run the job in an environment close to cron's:
env -i PATH=/usr/bin:/bin /usr/local/bin/nightly-backup.sh; echo "exit=$?"
command -v notify-admin          # shows /usr/local/bin/notify-admin

Fixes (any one, then re-verify): reference the tool by absolute path (/usr/local/bin/notify-admin), or set an explicit PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top of the script. Also add >> /var/log/nightly-backup.log 2>&1 to the crontab line so future failures leave evidence instead of disappearing.

Use an advisory lock that is tied to the process and released automatically if it dies, not a PID file (stale PID files are exactly the failure mode that bites at 02:00).

#!/bin/bash
set -euo pipefail
exec 9>/var/lock/nightly-backup.lock
if ! flock -n 9; then
    logger -t nightly-backup "previous run still active, skipping"
    exit 0
fi

Reasoning:

• Opening fd 9 and calling flock -n 9 makes the lock non-blocking: if another instance holds it, the new one exits immediately instead of queueing a second 40 GB tar on top of the first.
• Holding the lock on an open file descriptor means the kernel drops it when the script exits for any reason - crash, kill, reboot - so it can never go stale.
• exit 0 rather than a failure code is a deliberate choice for a skip: cron's mail should be reserved for real failures. If the business needs to know a run was skipped, send that through logger/the notification path instead of a non-zero exit.
• The file itself lives in /var/lock so it is on the standard lock location and cleaned by the normal boot process.
• The same lock works for manual runs: an operator invoking the script at 02:01 gets the skip message rather than corrupting the archive.

Overlap and false success are two symptoms of the same sloppiness: no error handling and no atomic output.

#!/bin/bash
set -euo pipefail                      # stop on error, undefined var, pipe failure
trap 'rm -f "$TMP"' EXIT
dest=/backup/home-$(date +%F).tar.gz
TMP="$dest.partial.$$"

if ! tar -czf "$TMP" -C / home; then
    logger -t nightly-backup "tar failed for $dest"
    exit 1                             # cron now reports a real failure
fi
mv -f "$TMP" "$dest"                   # atomic publish
/usr/local/bin/notify-admin "backup ok: $dest"

What each change buys:

• set -euo pipefail removes the masking effect of the trailing echo; any failing command now terminates the script with a non-zero status, which cron reports.
• Writing to $dest.partial.$$ and renaming at the end means a killed or overlapping run can never leave a truncated file at the real path - the rename is atomic on the same filesystem.
• The trap cleans up a partial file called the script out of the way.
• -C / with relative paths avoids embedding the leading / that makes tar emit 'Removing leading /' warnings and keeps the archive restorable with tar -C / .
• The lock from the previous task prevents overlap in the first place; the temp-file-and-rename pattern is the second line of defence.
• Log the destination and byte size with timestamps; a 0-byte or 20-byte archive should be visible in the log the next morning.