Emergency mode after SAN re-presentation plus an old default kernel
An Oracle Linux 7 server with root on LVM reboots to emergency mode after a SAN LUN holding /home was re-presented. The console shows 'Failed to mount /home' and 'Dependency failed for Local File Systems'; journalctl -xb shows a timeout waiting for the device with the UUID in /etc/fstab. Root and /var mount normally. blkid shows the /home filesystem now has a new UUID, and the data is intact. /boot is 95% full with six installed kernels. GRUB has GRUB_DEFAULT=saved, and /boot/grub2/grubenv still points to 4.14.35-2047.el7uek, although 4.14.35-2102.el7uek was installed two weeks ago and is the newest. The change window is 30 minutes; the server has no console access beyond the serial emergency shell, and the application team insists on keeping the same mount point and data.
1. What is the fastest safe path from the emergency shell to a multi-user boot without reformatting /home or losing data?
Read the worked answer
Recovery reasoning. The data is present; the only broken link is the identifier in fstab. Do not reformat or recreate the filesystem. At the emergency shell, run mount -o remount,rw /, then identify the device from blkid or lsblk -f and confirm it is the /home LUN, not a snapshot or another LUN of similar size. Edit /etc/fstab, replace only the UUID field, and preserve the mount options. Before rebooting, test with mount -a or mount /home; if it mounts, systemctl daemon-reload is harmless, then reboot. If the mount fails, inspect dmesg for multipath or XFS errors and do not force it. Because the application needs /home, nofail alone would hide the problem on the next boot, but adding x-systemd.device-timeout=30s prevents an indefinite emergency wait. Keep a copy with cp /etc/fstab /etc/fstab.bak. The 30-minute window is enough only if the LUN is visible; verify multipath -ll and lsblk first. If it is not visible, comment the /home line, boot multi-user, and fix storage separately.
2. How would you make /home resilient to future LUN re-presentations, and why is a UUID alone a weak choice on SAN storage?
Read the worked answer
Stable naming. A UUID is filesystem metadata, so re-presenting or restoring a LUN can change it; the UUID is stable only while the filesystem is unchanged. On SAN storage, prefer a name tied to the storage identity: /dev/disk/by-id/... for a filesystem directly on a LUN, or an LVM logical volume or multipath alias whose wwid is bound in multipath.conf. by-path is less stable because it changes with HBA and path. For /home, use the multipath by-id link or LVM LV path, and add x-systemd.device-timeout=30s; consider nofail only for non-critical filesystems. Verify with udevadm info and ls -l /dev/disk/by-id. Document the wwid in the runbook so storage re-presents the same LUN with the same identity. Do not make blkid after every change the process; that is a repair, not a design. Test the change by rebooting in a maintenance window, and keep the old fstab entry commented for audit. If the LUN is multipathed, also confirm the alias in multipath.conf and the wwid match before trusting the name.
3. The default kernel is old and /boot is nearly full. Describe the correct sequence to set the newest UEK as default and reclaim space without breaking the running system.
Read the worked answer
Kernel and /boot order. First free space, because grub2-mkconfig and a new initramfs need room. Confirm the running kernel with uname -r and never remove it. Install yum-utils if needed, then package-cleanup --oldkernels --count=2, which preserves the running kernel, or remove specific old kernel-uek packages. Check /etc/yum.conf for installonly_limit and set it to 3 if it is 0. Then list entries with grubby --info=ALL or awk -F\' '/^menuentry/ {print NR-1, $2}' /boot/grub2/grub.cfg and set the newest UEK by title or index with grub2-set-default. Make the configuration persistent with grub2-mkconfig -o /boot/grub2/grub.cfg on BIOS, or the EFI path if UEFI, and verify grub2-editenv list and grubby --default-kernel. Rebooting is required to run the new kernel; the 30-minute window must include a boot test and application validation. Keeping two kernels is enough for rollback; six is not a safety margin, it is a /boot outage risk.