Emergency Mode After an fstab Edit

1. Make root writable - the emergency shell inherits the read-only root:

mount -o remount,rw /

Without this, edits to /etc/fstab appear to succeed and are lost or fail with EROFS.

1. Fix the entry. For a remote mount that may be unavailable at boot:

nfsserver:/export/archive /mnt/archive nfs nofail,_netdev,x-systemd.automount,x-systemd.mount-timeout=15s 0 0

nofail means a failed mount does not block the boot. _netdev orders the mount after the network is up. x-systemd.automount defers the actual mount to first access, so boot never waits on the remote at all, and x-systemd.mount-timeout caps how long a later access can hang.

1. systemctl daemon-reload. This is the step people skip. systemd generates mount units from fstab when it loads it, so an edited fstab without a reload leaves the old, failing unit in the manager's view - and the next systemctl default re-enters the same failure.

2. systemctl default (or isolate multi-user.target) to continue the boot.

Do not run mount -a here: with the remote still down it re-triggers the hang you are trying to escape. Do not comment the line out permanently - you would silently lose monitoring of a mount that production expects.

Against, with a caveat. Debt-free recovery is the priority, so getting the host up matters - but deleting the line trades a 15-minute outage for a silent functional gap: anything writing to /mnt/archive starts writing to the empty underlying local directory instead, and the archive volume fills with files the NFS server never sees. That is worse than a clean failure and much harder to detect. The two-line boot-option fix costs the same amount of console time and keeps the intent of the mount visible.

If the entry really must go, record it in the change ticket and add a monitoring check on the mount before the change is closed.

Verify before the next reboot:

findmnt --verify --verbose     # parses fstab, reports syntax and unreachable-source problems
systemctl list-units --type=mount --all | grep archive
systemctl list-units --type=automount --all
systemd-analyze verify /etc/fstab   # not applicable to fstab; use findmnt instead

Then test the deferred path deliberately: with the NFS server still down, ls /mnt/archive should return promptly with an error rather than hanging the shell, and systemctl status mnt-archive.automount should show it waiting.

Finally, fix the golden image - this entry is about to be baked into twenty more hosts, and a boot-blocking remote mount is a fleet-wide outage waiting for one NFS blip.

This is the real cost of nofail: it converts a loud boot failure into an invisible runtime one.

Alert on mount state directly, not on the filesystem being merely present:

findmnt --target /mnt/archive --evaluate --json
systemctl is-active mnt-archive.mount
stat -f -c %T /mnt/archive     # 'nfs' vs 'xfs'/'ext2/ext3' tells you if the real mount is there

The stat -f check is the one that matters: if the local underlying directory is unmounted-but-present, findmnt --target reports the parent filesystem, so a naive [ -d /mnt/archive ] check passes happily while nothing is mounted.

Design the check so the probe itself cannot hang: run it with timeout 20, because an automount attempt against a dead server will stall until x-systemd.mount-timeout expires.

Then close the loop with the application owner: a job that writes to /mnt/archive should fail loudly on ENOTCONN/ESTALE rather than creating files in an empty local directory, and the runbook should say 'check the mount before investigating the job'. An alert that only fires after the disk fills with misplaced archives is not an alert.