Goodbye mkinitcpio, hello dracut! #

This is part one (of two) of my journey to remote unlocking encrypted (LUKS) root partitions using SSH during early boot. To achieve this, as a prerequisite, I needed to switch from mkinitcpio to dracut for building the initramfs on one of my systems. So I thought, I might share the few steps I needed to take, to successfully switch over to dracut on my archlinux system.

If you just want the details of what files need to be modified or created, which packages to be installed and stuff, you can skip to this section here. You can find the second part here: Bootstrapping OpenSSH: Remote Disk Decryption.

What an initramfs? #

If you already know, you can of course simply skip this section. Or read it and correct me where I am wrong.

Essentially, the initramfs is a RAM disk, i.e. filesystem stored in memory, which is used to initialize the system to a point, where it can access the root filesystem. To achieve this, the kernel, which gets started by the bootloader, is responsible of unpacking a contained initramfs image and bootstrap the system. This initramfs image contains all the tooling required to e.g. unlock the filesystem in case of full disk encryption (FDE), load required drivers to e.g. read keyboard events, and load other components as configured (e.g. the networking/IP stack).

This RAM disk is stored as a binary file at a place where your bootloader can find it. This depends on your boot method and bootloader. Using UEFI and Grub as an example, this file is simply located in the root of your EFI boot partition, which usually gets mounted as /boot. The bootloader is then told, which image(s) to load at boot time.

Tools like dracut or mkinitcpio can be used to configure the initramfs image. They essentially create the filesystem, add all the files to it, add some metadata, and pack them into an image file. On linux, the command lsinitrd can be used, to inspect such an image. It displays information about the early CPIO (copy in, copy out) files, the version (e.g. dracut), modules to load and the entire initramfs filetree.

Why dracut? #

mkinitcpio is fine, but not widely adopted. Most distributions, including Debian, Ubuntu, Fedora, RHEL and many more, use dracut. This of course reflects on the amount of information and solutions available for each.

For me, the main deciding factor was the ease of configuration. While neither option is really all that complex, dracut is easier to configure and needing to administer multiple systems means I don’t want to maintain multiple solutions. So I wanted a common basis for all my systems. Therefore my decision fell on dracut.

First build #

Now, regarding the actuall steps required to switch from mkinitcpio to dracut: firstly, we need to install dracut. On most systems it is typical, that before installing new packages, one should update the system. In this case, if the update pulled in a newer kernel version or other package version which required rebuilding the initramfs, then the system should be rebooted afterwards to make the first initramfs build using dracut will actually use the latest installed kernel and not the active but out-of-date version.

Once rebooted (if necessary), we simply install our distributions dracut-package (e.g. dracut) using the package manager. We won’t uninstall mkinitcpio yet, only once we have thoroughly tested that dracut actually works. Also, only then will we configure pacman to use dracut instead of mkinitcpio to rebuild the initramfs on kernel updates.

The main configuration file for dracut is /etc/dracut.conf, but we won’t edit that one directly. Instead, we’ll work on our own file in /etc/dracut.d/, e.g. myflags.conf. This allows for distribution maintainers to update /etc/dracut.conf if needed on system updates, but also allow us to declutter our changes and keep them sorted. For now, we’ll simply add the following lines to our /etc/dracut.d/myflags.conf:

hostonly="yes"
hostonly_cmdline="no"

The second line prevents dracut from storing the kernel command line in the initramfs. This might not be useful for everyone. If you are configuring the command line using the bootloader (e.g. Grub), then there is no need for this. AFAIK, setting it to yes can also cause some issues on some systems, especially on non-systemd ones, but don’t quote me on that. Also, dracut’s command line parameters take precedence over configuration files.

The first line on the other hand, is also important for SSHD specifically. While simply telling dracut to build the initramfs image only for this device instead of a generic host, meaning it shouldn’t be used while chrooted, this is also important on some systems and versions of dracut to ensure that the root user can actually be logged into via SSH. Wthout this option, the /etc/shadow file, which contains the hashes of user passwords, might not be created/included in the image. Therefore, once you have built your initramfs image, you should verify whether your image contains an etc/shadow file (note the absence of a leading slash!) and that it contains a line starting with root: followed by either a hash (starting with $), or an asterisk (*) using the following command:

lsinitrd /boot/IMAGE_NAME.img -f etc/passwd

Now, simply running sudo dracut -f -v should build a new initramfs for the current kernel. The -f is used to force dracut to overwrite the existing initramfs image, -v for verbosity (which is the default on some systems). On some systems the command might fail the first time. In that case, you might need to run sudo dracut -f -v --regenerate-all, but your mileage may vary. You can also specify a path to overwrite a specific image file by simply appending it to the command above. If you didn’t reboot after a kernel update, you could also specify --kver ... to build the image for a specific kernel version.

Reboot the system to see whether your bootloader was able to detect the new initramfs image, and select it. If you use FDE for your root partition, enter the passphrase. Now, if your system boots correctly, dracut works and you can continue with the next steps. But if nothing seems to happen or you don’t get any passphrase prompt (but expected one, of course), you might want to press Escape to get more detailed information about the boot process. This is not necessary if your kernel command line doesn’t include quiet. To get even more detailed information, you can specify the kernel command line options rd.shell, rd.debug and log_buf_len=1M. This helped me to discover a boot loop which didn’t make sense and was fixed by simply rebuilding the initramfs. But for more concrete steps, consult the documentation or ask your system administrator (that’s a joke).

Automatic updates #

This section assumes a system using the pacman package manager (like archlinux). For other systems, similar solutions probably exist using hooks for the respective package managers. These steps are based on this section in the archlinux wiki entry on dracut.

For starters, we’ll create two executable scripts to be run automatically by pacman using so-called hooks. The first one simply automatically copies the vmlinuz binary (single binary that contains the kernel) to /boot and calls dracut to build both the new initramfs as well as a generic fallback image for the latest kernel version. Simply create the file /usr/local/bin/dracut-install.sh (the exact path doesn’t matter but must match the ones used in the hooks further below) with the following contents:

#!/usr/bin/env bash
args=('--force' '--no-hostonly-cmdline')

while read -r line; do
	if [[ "$line" == 'usr/lib/modules/'+([^/])'/pkgbase' ]]; then
		read -r pkgbase < "/${line}"
		kver="${line#'usr/lib/modules/'}"
		kver="${kver%'/pkgbase'}"

		install -Dm0644 "/${line%'/pkgbase'}/vmlinuz" "/boot/vmlinuz-${pkgbase}"
		dracut "${args[@]}" --hostonly "/boot/initramfs-${pkgbase}.img" --kver "$kver"
		dracut "${args[@]}" --no-hostonly "/boot/initramfs-${pkgbase}-fallback.img" --kver "$kver"
	fi
done

The second file, /usr/loca/bin/dracut-remove.sh, simply removes old images. If you want to retain the previous version, simply modify this script to do so. It should look something like this:

#!/usr/bin/env bash
while read -r line; do
	if [[ "$line" == 'usr/lib/modules/'+([^/])'/pkgbase' ]]; then
		read -r pkgbase < "/${line}"
		rm -f "/boot/vmlinuz-${pkgbase}" "/boot/initramfs-${pkgbase}.img" "/boot/initramfs-${pkgbase}-fallback.img"
	fi
done

Don’t forget to make both files executable:

chmod +x /usr/local/bin/dracut-{install,remove}.sh

Now all that’s left to do, is creating the pacman hooks themself, one for each script. Hooks are placed in /etc/pacman.d/hooks. This directory might not be created initially, so ensure it exists. Then just create the pacman hook /etc/pacman.d/hooks/90-dracut-install.hook and add the following to it:

[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Target = usr/lib/modules/*/pkgbase

[Action]
Description = Updating linux initcpios (with dracut!)...
When = PostTransaction
Exec = /usr/local/bin/dracut-install.sh
Depends = dracut
NeedsTargets

Followed by /etc/pacman.d/hooks/60-dracut-remove.hook:

[Trigger]
Type = Path
Operation = Remove
Target = usr/lib/modules/*/pkgbase

[Action]
Description = Removing linux initcpios...
When = PreTransaction
Exec = /usr/local/bin/dracut-remove.sh
NeedsTargets

That’s it. A fairly straight forward process. If you are curious about why I needed this in the first place, then you can find the second part here: Bootstrapping OpenSSH: Remote Disk Decryption.