Nightly archive works interactively, fails from root cron
On an Ubuntu 22.04 host, /usr/local/bin/nightly-backup.sh must create /var/backups/www-YYYY-MM-DD.tar.gz and upload it with the AWS CLI to object storage. The script starts with #!/bin/sh, uses [[ ... ]] tests and a Bash array to list web roots, and calls aws without an absolute path. Run manually as root with bash /usr/local/bin/nightly-backup.sh, it finishes in 40 seconds and writes a 1.2 GB archive. The root crontab entry is 30 2 * * * /usr/local/bin/nightly-backup.sh >> /var/log/nightly-backup.log 2>&1. For three nights no archive appears, and the log contains only: /usr/local/bin/nightly-backup.sh: 14: Syntax error: "(" unexpected. The previous administrator moved the script from /root/scripts to /usr/local/bin and edited the first line; the script was tested only with bash. The AWS CLI is installed at /usr/local/bin/aws. No changes to /etc/crontab or /etc/environment are permitted; the fix must live with the job.
1. Name the immediate cause of the syntax error and the second failure that will appear after the shebang is corrected. Give a command that proves each one before you edit anything.
Read the worked answer
Two separate faults are stacked. First, /bin/sh on Ubuntu is dash, not Bash. The shebang #!/bin/sh makes cron's kernel exec use dash, and dash rejects Bash-only [[ ... ]] and array syntax. Running bash script bypasses the shebang entirely, which is why manual testing passed. Prove it before editing with readlink -f /bin/sh (expect /usr/bin/dash) and dash -n /usr/local/bin/nightly-backup.sh (expect a syntax error around line 14). Second, cron starts jobs with a minimal PATH, typically /usr/bin:/bin; aws is in /usr/local/bin. After fixing the shebang, the upload will fail with aws: not found. Simulate cron's environment with env -i PATH=/usr/bin:/bin /bin/sh -c 'command -v aws'; it prints nothing, while your interactive shell prints /usr/local/bin/aws. The log line already in the file confirms the first fault; do not edit until you have captured both outputs.
2. Rewrite the script header and cron entry so the job runs with Bash and a deterministic environment, and add safeguards that turn silent failure into a visible one. Explain why each change matters.
Read the worked answer
Use Bash explicitly and make the environment explicit; do not depend on the invoker's PATH. Header:
#!/bin/bash
set -Eeuo pipefail
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
umask 0077
set -e makes a failed tar, aws, or mv stop the run; pipefail prevents a pipeline from hiding an earlier failure; set -u catches misspelled variables. Root crontab:
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[email protected]
30 2 * * * flock -n /run/nightly-backup.lock /usr/local/bin/nightly-backup.sh >>/var/log/nightly-backup.log 2>&1 || logger -t nightly-backup 'backup failed'
Reasoning: the shebang no longer depends on /bin/sh; the cron PATH contains the AWS CLI even if cron ignores the script's exported PATH until after exec (the script sets it again); MAILTO gives mail if an MTA exists. flock prevents overlap. The || logger is only a backstop: it logs a non-zero exit to the journal, but the script itself should also log stage names and verify the archive size and upload exit status. Validate with bash -n, then run once under env -i with the same PATH.
3. An operator sometimes starts the script manually at 02:35 while cron is running. Explain how to prevent concurrent uploads, and why a PID file is riskier than flock. What should the script do when the lock is held?
Read the worked answer
Use a kernel advisory lock on a file under /run, not a hand-built PID file. In the script:
exec 9>/run/nightly-backup.lock
if ! flock -n 9; then
logger -t nightly-backup 'another run holds the lock; skipping'
exit 75
fi
flock is released automatically when the process exits, including SIGKILL, so there is no stale lock to clean up. A PID file is riskier: after a crash the file remains, and a recycled PID can make kill -0 report a live process, so you must store start time or use flock anyway. If the manual operator should wait rather than skip, use flock -w 1800 9; cron can use -n to avoid piling up. Exit 75 is EX_TEMPFAIL, which makes the condition visible to cron or monitoring without pretending the backup succeeded. Do not unlink the lock file; unlink/create races can defeat locking. Also make the archive name date-based and write to a temporary name then mv so a killed run cannot leave a partial file that looks valid.