Custom Service Starts Before NFS Mount on Boot
A custom Python application runs as myapp.service on a Linux server. It reads and writes /mnt/nfs/data, an NFS mount defined in /etc/fstab with _netdev. The service starts correctly when you run systemctl start myapp after boot, but after a reboot it fails with Permission denied writing to /mnt/nfs/data. journalctl -u myapp shows the service started at 12 seconds after boot; the NFS mount completed at 18 seconds. The unit file currently has only [Unit] Description=MyApp, [Service] ExecStart=/opt/myapp/run.sh, Restart=on-failure, and [Install] WantedBy=multi-user.target. The server has no local fallback; if the NFS mount is unavailable, the app should not start and should retry later. You need to fix the ordering and dependency without masking real failures.
1. Diagnose the boot-order failure and explain why `Restart=on-failure` did not save the service.
Read the worked answer
myapp starts before the NFS mount is active. The mount unit for /mnt/nfs/data is ordered after network-online.target due to _netdev, but myapp has no ordering dependency, so systemd starts it as soon as multi-user.target is reached. The app writes to an empty mountpoint, gets Permission denied or writes to the local filesystem, and exits. Restart=on-failure may retry, but it does not wait for the mount; if the restart limit is hit or the exit code is considered clean, it stops. Also, a restart after the mount appears could succeed, but the logs show the initial failure. The fix is an ordering and requirement dependency, not just restart.
2. Show the unit file changes you would make, including dependency and retry behavior, and explain each directive.
Read the worked answer
Use a drop-in or edit the unit:
[Unit]
Description=MyApp
After=network-online.target remote-fs.target
Wants=network-online.target
RequiresMountsFor=/mnt/nfs/data
[Service]
ExecStart=/opt/myapp/run.sh
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
RequiresMountsFor= adds a dependency on the mount unit and ensures it is active before myapp starts; it also makes myapp stop if the mount goes away. After=remote-fs.target is a coarse ordering; RequiresMountsFor is more precise. Wants=network-online.target pulls in the target, but you must ensure a wait-online service is enabled. Keep Restart=on-failure with a delay to handle transient NFS outages. If the mount fails permanently, the service should remain down rather than write to local disk.
3. How would you verify the fix and what would you check if the service still fails after reboot?
Read the worked answer
Validate with systemd-analyze verify myapp.service and systemd-analyze critical-chain myapp.service. Check dependencies with systemctl list-dependencies myapp.service. After reboot, inspect journalctl -b -u myapp, systemctl status mnt-nfs-data.mount, mount | grep nfs, and systemctl status remote-fs.target. If it still fails, check NFS server export permissions, /etc/fstab options (_netdev, nofail), SELinux/AppArmor, and whether network-online.target actually waited. systemd-analyze plot > boot.svg can show ordering. Test by rebooting twice.