Initramfs shell after a controller swap

Two different mount stages are being confused.

• GRUB loads the kernel and initrd using firmware disk services, so the boot loader works even though Linux has no driver for the new controller.
• Inside the initramfs, the root filesystem is mounted from the root= parameter on the kernel command line, not from /etc/fstab. Editing fstab changes nothing here.
• A wrong root= entry in fstab would produce a different symptom: root mounts, then systemd fails on local-fs.target and drops to emergency mode, where systemctl --failed and journalctl -xb name the failing mount.

Diagnosis at the busybox prompt:

1. cat /proc/cmdline - confirm the kernel really was given root=UUID=1c9f....
2. cat /proc/partitions and ls /dev/sd* - if only loop/ram devices exist, the kernel never saw the disks: a missing driver, not a UUID problem.
3. blkid - if a partition appears but with a different UUID, the cause is the identifier or a different disk, not the driver.
4. modprobe ahci; blkid - if the device appears only after the manual load, the module is present in the initrd but is not being loaded, or is absent and must be added. dmesg | tail shows the probe result.

Note that this initramfs shell is busybox: no systemd, no udev helpers beyond what the image ships, and blkid only knows what exists in /dev.

Root cause: the initrd was generated while the disks were on the LSI card, so it contains the module set needed at build time. mkinitramfs with MODULES=dep in /etc/initramfs-tools/initramfs.conf includes only the drivers needed to reach the root device as seen on the building machine; mpt3sas made the cut, ahci/sd_mod did not. That is why rebuilding with -k all looked correct and changed nothing.

Debian:

# echo ahci >> /etc/initramfs-tools/modules
# update-initramfs -u -k all
# lsinitramfs /boot/initrd.img-$(uname -r) | grep -i ahci   # verify, don't assume

No update-grub is required: root= did not change and GRUB is fine.

RHEL family:

# echo 'add_drivers+=" ahci "' > /etc/dracut.conf.d/99-ahci.conf
# dracut -f --regenerate-all

The alternative - setting MODULES=most - also works but inflates every initrd on the fleet. Prefer the explicit module list: it documents why the module is there and survives the next kernel upgrade. Rebuilding must happen on a machine that can see the new hardware layout, or with the module listed explicitly, otherwise the same trap repeats at the next kernel upgrade.

Root is now mounted, so the initramfs stage is over; this is a unit-startup failure in the real system, almost always a mount that systemd cannot satisfy. The dropped-to-emergency message normally names it (Failed to mount /..., Dependency failed for Local File System).

Checks in order:

1. systemctl --failed - names the failed unit, typically a .mount generated from /etc/fstab.
2. journalctl -xb | grep -i -e mount -e uuid - the exact UUID or device the system wanted.
3. mount -a after systemctl default (or from the emergency shell) - prints the failure directly.

Typical causes on this host: a swap or data-disk fstab entry still pointing at the UUID of a partition that no longer exists after the controller change, or the /dev/sda1-style edit the colleague proposed being pasted into the wrong field. Permanent fixes: correct the UUID with blkid, or add nofail (plus x-systemd.device-timeout=) for non-essential mounts so a missing disk degrades instead of blocking boot. Do not mask the unit - fix or annotate the fstab entry, because the same fault returns at the next boot.