Arch boot recovery

2023-06-08 • 3 minute read • problem • linux

Last time I left home, I upgraded both identical laptops. The one I took with me worked without any problems. Today I was puzzled to find the second laptop unable to boot.

...
Waiting 10 seconds for device [...]
ERROR: mounting '/dev/mapper/luksdev' on real root

GRUB dropped the computer to emergency shell, but it wasn’t helpful. The keyboard didn’t work, including Ctrl-Alt-Del. I sighed, as I thought I’d be doing data recovery (faster than downloading the backup) from a live boot, then reinstalling, and recovering the files.

I was not asked for the disk encryption password — the partition couldn’t be unlocked. As GRUB was trying to mount from /dev/mapper/, something went wrong with LUKS.

With the emergency shell unusable, I tried booting from a different entry, without any luck. Next, I grabbed my keyring with USB drives. It just so happened, that the first one I tried, that usually has Fedora on it, booted in to Arch instead.

With the pain of the reinstall being a time sink, I thought if maybe, this time, I could recover the system successfully. Could the boot environment be fixed without any serious GRUB magician skills?

Already searched the web for the boot errors, I had a few commands to try for the emergency shell, just chroot in, and go:

# lsblk
...
nvme0n1     259:0    0  1.8T  0 disk
├─nvme0n1p1 259:1    0  511M  0 part
└─nvme0n1p2 259:2    0  1.8T  0 part

Judging by the size, the first one is boot, and the second root.

# cryptsetup luksOpen /dev/nvme0n1p2 main
# mount /dev/mapper/main /mnt
# mount /dev/nvme0n1p1 /mnt/boot
mount: boot: mount point does not exist.
# mkdir /mnt/boot
# mount /dev/nvme0n1p1 /mnt/boot
# ls /mnt
boot @ @home @log @pkg @.snapshots

I forgot to specify the Btrfs subvolume…

# umount /mnt/boot
# rm -d /mnt/boot
# umount /mnt
^R main
# mount /dev/mapper/main -o subvolume=@ /mnt
# ls /mnt
bin boot dev etc f home lib lib64 mnt opt proc r root run sbin srv sys tmp usr var
^R mnt/boot
# mount /dev/nvme0n1p1 /mnt/boot

Yet, I did not try the commands I had in front of me. Maybe I could fix this easier. I recall not reading the output for updates on the failing laptop. Maybe something went wrong.

# iwctl station wlan0 scan
# iwctl station wlan0 connect '...'
# arch-chroot /mnt
# pacman -Syu
* mkinitpcio stuff, the slowest part in upgrades *
#
^D
# reboot

It has been a few days since I last updated. As updating again should rebuild the boot environment anyway, I tried it. Looking at the logs, I was correct.

The laptop booted normally after the upgrade. I guess some problems can be solved by not diving deep in to it. If there wasn’t an upgrade, I would’ve tried reinstalling the kernel package.