Google IT Automation with Python: Killing the Messenger

Killing the Messenger

A Python maintenance script runs every five minutes from cron as root on 40 Linux web hosts. It is meant to terminate stale worker.py processes. Cron invokes /usr/bin/python3 /opt/scripts/cleanup.py worker.py. The script runs subprocess.run("ps -eo pid,cmd | grep worker.py | awk '{print $1}'", shell=True, capture_output=True, text=True) and sends SIGTERM to every PID in stdout. After a deployment, operators report two symptoms. On host A, the cleanup script itself disappears soon after starting, and stale workers survive; cron logs show no traceback. On host B, a developer's interactive vim worker.py session was killed, and a tail -f worker.py.log process was also terminated. The host runs systemd, and the worker command line is /usr/bin/python3 /opt/app/worker.py --queue payments. The script does not check whether a PID still exists before signaling, does not compare start times, and runs with the same UID as the workers. A quick pgrep -f worker.py returns many PIDs, including the shell that launched the pipeline.

1. Diagnose both host A and host B symptoms from the pipeline and pattern-matching behavior.

Read the worked answer

Host A: the cleanup script's own command line is /usr/bin/python3 /opt/scripts/cleanup.py worker.py, so ps output matches the substring worker.py and includes the cleanup process itself. The pipeline then emits its own PID and the script sends SIGTERM to itself before it finishes terminating workers. That explains a silent disappearance and surviving stale workers.

Host B: the pattern is a substring match over the full command line, not an executable match. vim worker.py and tail -f worker.py.log both contain the pattern. The parent sh -c ps ... grep worker.py ... also contains it. The script signals every emitted PID without checking argv, ownership, start time, or whether the process is even the target.

There is no traceback because SIGTERM is a signal, not a Python exception in the signaled process. A pgrep -f test confirms the broad match; it does not prove that the matched processes are workers.

2. Design a safer termination routine. Use exact argv matching, exclude the cleanup process and its ancestors, and avoid signaling unrelated processes.

Read the worked answer

Read /proc/<pid>/cmdline or use psutil, compare the argv vector exactly, and skip self, parent, and the current process group. Do not use shell=True for the discovery step. Signal only after re-checking identity to reduce PID-reuse risk.

import os, signal, psutil

TARGET_ARGV = ['/usr/bin/python3', '/opt/app/worker.py', '--queue', 'payments']
me = os.getpid()
protected = {me, os.getppid()}

for proc in psutil.process_iter(['pid', 'ppid', 'cmdline', 'create_time']):
    pid = proc.info['pid']
    argv = proc.info['cmdline'] or []
    if pid in protected or pid == os.getpgid(0):
        continue
    if argv[:3] != TARGET_ARGV[:3]:
        continue
    if '--queue' in argv and argv[argv.index('--queue') + 1] != 'payments':
        continue
    try:
        proc.terminate()
    except psutil.NoSuchProcess:
        pass

The TARGET_ARGV comparison prevents vim worker.py and tail -f worker.py.log from matching. Excluding self and parent prevents the host A failure. For production, prefer a systemd unit or cgroup-based kill, because systemd can terminate the exact service's process group without relying on command-line matching.

3. How would you test this routine without root and without risking the test runner's real processes?

Read the worked answer

Refactor discovery into a function that accepts an iterable of process records and a signal function. Unit-test that function with a crafted list containing: the cleanup PID, the parent PID, a true worker, a vim worker.py decoy, a tail decoy, and a process with a similar prefix but a different queue. Inject nohup-style signal collectors and assert exactly the true worker PID is signaled.

Then run one integration test in a disposable container as a non-root user: start two fake workers, one decoy sleep 600 with worker.py in an argument, run the routine, and assert the workers exited while the decoy and test harness survived. Never point the integration test at the shared host; a mistaken pattern on a production host could kill the same session running the test.