Google IT Automation with Python: The Backup That Only Checked the Folder Existed

The Backup That Only Checked the Folder Existed

A nightly Python 3.12 backup job copies /srv/app/data (1,204 files, about 48 GB) to /backup/appdata using shutil.copytree(source, dest, dirs_exist_ok=True), wrapped in a single try/except OSError that logs one warning and continues to write a marker file, then exits 0. Monitoring alerts only if the process exits nonzero or if /backup/appdata does not exist. Last week a restore drill found only 312 files in the backup; 19 of them were zero bytes, and an entire attachments/ subdirectory was missing. The source filesystem had hit a quota for several hours, so some writes failed with ENOSPC. The destination already contained the previous night's files, and copytree overwrote them in place. The marker file was written after the failed copy, so it existed. The backup disk itself has 60 GB free. The team wants backups that fail loudly, avoid publishing partial files, and produce evidence that a restore will work.

1. Why did monitoring stay green, and why is the resulting directory unsafe even though some files are present?

Read the worked answer

The script catches OSError, logs one warning, and keeps going. It then writes the success marker and exits 0, so both monitored conditions are satisfied: the exit code is zero and /backup/appdata exists. The warning is not an alertable signal.

The directory is unsafe because copytree wrote directly into the live backup path with dirs_exist_ok=True. A failure partway through leaves a mixture of old files, new files, zero-byte files, and missing subdirectories. There is no manifest, file count, byte count, or checksum to distinguish a complete backup from a partial one. The next restore could silently return stale data from an earlier night alongside truncated new files. The existence of the directory is not evidence of recoverability.

2. Describe a safer backup publication process, including staging, verification, and failure signaling.

Read the worked answer

Write to a per-run staging directory on the same filesystem, verify, then publish atomically. Do not overwrite the previous good backup in place.

Before copying, check free space with shutil.disk_usage and require source size plus a safety margin. 60 GB free for a 48 GB source is tight once the previous backup is retained.
Copy into /backup/appdata.staging.<run-id> using copytree without dirs_exist_ok on an empty directory. Let any OSError propagate; do not catch and continue.
Compare source and staging: file count, total bytes, and a SHA-256 manifest. Optionally re-read a sample. Treat mismatches as hard failures.
Publish only after verification: rename the staging directory to a versioned path such as /backup/appdata.2025-06-10, then atomically update a current symlink. Keep the previous version for rollback.
Write a success marker containing run id, counts, bytes, manifest hash, and timestamp only after publication. On any failure, remove the staging directory, leave the previous backup untouched, and exit nonzero with a structured error.

A failed run should page someone or open a ticket; a warning line in a log is not a failure signal.

3. Write a failure-injection test that proves a quota error cannot publish a partial backup or overwrite the last good one.

Read the worked answer

Use tmp_path for source, backup root, and a fake previous backup. Monkeypatch the copy function to raise OSError(errno.ENOSPC, 'No space left on device') on the 313th file. Assert that the run exits nonzero, that the success marker is unchanged, that the staging directory is removed, and that the previous backup's files are byte-identical.

import errno, shutil, pytest

def test_enospc_does_not_publish(monkeypatch, tmp_path):
    src = tmp_path / 'src'
    src.mkdir()
    for i in range(400):
        (src / f'f{i}').write_text(str(i))
    backup = tmp_path / 'backup'
    old = backup / 'appdata'
    old.mkdir(parents=True)
    (old / 'known-good').write_text('keep me')

    real_copy = shutil.copy2
    count = {'n': 0}
    def failing_copy(a, b, *args, **kwargs):
        count['n'] += 1
        if count['n'] == 313:
            raise OSError(errno.ENOSPC, 'No space left on device')
        return real_copy(a, b, *args, **kwargs)

    monkeypatch.setattr(shutil, 'copy2', failing_copy)
    rc = run_backup(src, backup)
    assert rc != 0
    assert (old / 'known-good').read_text() == 'keep me'
    assert not list(backup.glob('appdata.staging.*'))

Add a manifest-mismatch test and a test that the success marker is not written before verification. The tests should assert observable filesystem state, not merely that a warning was logged.