LFCS: reportd restart loop after reboot: ordering versus directory readiness

reportd restart loop after reboot: ordering versus directory readiness

On an Ubuntu 22.04 host you deploy a Go binary at /opt/reportd/reportd with unit /etc/systemd/system/reportd.service: ExecStart=/opt/reportd/reportd --config /etc/reportd/config.yaml, User=reportd, After=network.target, Restart=always, RestartSec=5. Running that exact command line as root works. After a reboot the service enters a restart loop. journalctl -u reportd shows status=200/CHDIR together with a one-time "Failed to set up mount namespacing" message, then repeated connect: no route to host errors to a database at 10.60.4.20. The host gets its address over DHCP on a segment whose router needs about 12 seconds to become usable. /var/lib/reportd must exist and be owned by reportd, but it is currently created by hand during installs. The maintenance window forbids rebooting.

1. The journal contains two unrelated failure classes. Separate them and state, for each, the exact commands you would run on the live host to confirm the diagnosis.

Read the worked answer

Class A - status=200/CHDIR plus a namespacing complaint.

Exit code 200/CHDIR means systemd could not enter the unit's working directory before exec. Since the unit does not set WorkingDirectory=, the likely cause is a path referenced by the unit's directory/sandboxing options (or the program's default relative path) that does not exist at boot: /var/lib/reportd is created by hand, so on a fresh boot it is absent.

Confirm without rebooting:

systemctl show -p WorkingDirectory,StateDirectory,ReadWritePaths,RuntimeDirectory reportd
systemd-analyze verify /etc/systemd/system/reportd.service
ls -ld /var/lib/reportd

Then reproduce empirically: mv /var/lib/reportd /root/ and systemctl restart reportd; the same 200/CHDIR should return. Restore the directory. A passing diagnosis here must be reproduced, not assumed.

Class B - connect: no route to host.

After=network.target only orders the unit after the link-level configuration step. It says nothing about DHCP having completed or a default route existing. On a segment where the router needs ~12 s, the daemon starts first and fails.

Confirm with:

systemctl show -p After,Wants,Requires reportd
systemctl status systemd-networkd-wait-online 2>/dev/null || systemctl status NetworkManager-wait-online
journalctl -b -o short-monotonic -u reportd -u systemd-networkd

In the monotonic journal, reportd's first connect attempt should precede the DHCP lease line. That timeline, not the error text alone, is the proof.

2. Write the directive changes (as a drop-in) that fix both classes, and explain how you validate the ordering without rebooting and what would silently make your fix a no-op.

Read the worked answer

# /etc/systemd/system/reportd.service.d/override.conf
[Unit]
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
StateDirectory=reportd
Restart=on-failure
RestartSec=10

Reasoning:

Wants= + After=network-online.target delays start until the network is usable, which is the actual requirement; network.target was the wrong dependency.
StateDirectory=reportd makes systemd create /var/lib/reportd with the correct owner, and implicitly adds it to the unit's writable paths - this removes the hand-made-directory precondition instead of documenting it.
Restart=on-failure + StartLimit* stops a broken config from masking itself behind an infinite Restart=always loop, which is what hid the root cause here.

Validation:

systemd-analyze verify /etc/systemd/system/reportd.service
systemctl daemon-reload
systemctl show -p After,Wants,StateDirectory reportd
rm -rf /var/lib/reportd && systemctl restart reportd   # proves StateDirectory works
systemctl stop reportd; systemctl start reportd; systemctl status reportd

The silent no-op: network-online.target is only meaningful if some wait-online provider is enabled. Check with systemctl list-unit-files '*wait-online*' and confirm the one matching your network manager is enabled; otherwise the target is reached instantly and you have reordered nothing. Also note systemd-analyze verify will not catch a missing wait-online provider.