Thankfully, Pixiecore makes it easy to set up an iPXE server on NixOS, a task that could otherwise be quite complex.

The simplest iPXE server we can set up on NixOS can look like the following:

{ config, lib, pkgs, ... }:

{
  services.pixiecore = {
    enable = true;
    openFirewall = true;
    dhcpNoBind = true;
    kernel = "https://boot.netboot.xyz";
  };
}

In this configuration, we enable the Pixiecore service, set it to work along the DHCP server that already exists on our network and to boot the netboot.xyz iPXE image, a common option which gives you a selection of live installers on boot.

This may be enough for your environment but, as we’re running NixOS, we’d like to an image customized to our needs, having SSH access pre-configured, for example:

{ config, inputs, lib, pkgs, ... }:

let
  sys = inputs.nixos.lib.nixosSystem {
    system = "x86_64-linux";
    modules = [
      ({ config, pkgs, lib, modulesPath, ... }: {
        imports = [ (modulesPath + "/installer/netboot/netboot-minimal.nix") ];
        config = {
          services.openssh = {
            enable = true;
            openFirewall = true;

            settings = {
              PasswordAuthentication = false;
              KbdInteractiveAuthentication = false;
            };
          };

          users.users.root.openssh.authorizedKeys.keys = [
            "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI..."
          ];
        };
      })
    ];
  };

  build = sys.config.system.build;
in {
  services.pixiecore = {
    enable = true;
    openFirewall = true;
    dhcpNoBind = true; # Use existing DHCP server.

    mode = "boot";
    kernel = "${build.kernel}/bzImage";
    initrd = "${build.netbootRamdisk}/initrd";
    cmdLine = "init=${build.toplevel}/init loglevel=4";
    debug = true;
  };
}

The previous example does exactly that.

This now becomes very powerful and we’ve just starting using this on the lab computers we manage. We set the boot order to boot first from the hard disk and, if it’s not bootable, it will automatically boot from iPXE, still allowing remote access from the administrators and allowing us, if necessary, to redeploy its configuration with nixos-anywhere, as I’ll explain in a future blog post.

But we also want to be able to trigger an iPXE boot remotely, even if the current hard disk has a bootable configuration.

Initially, it wasn’t very obvious how we should be able to do this. One possibility would be to use efibootmgr to change the boot order accordingly, but it would make it more difficult to return to booting from local hard disk after the redeploy.

After some reflection, we settled on creating a special GRUB entry that boots from iPXE.

{ config, lib, pkgs, ... }:

{
  boot.loader = {
    efi.canTouchEfiVariables = true;
    grub = {
      enable = true;
      efiSupport = true;
      device = "nodev";

      extraFiles = { "ipxe.efi" = "${pkgs.ipxe}/ipxe.efi"; };
      extraEntries = ''
        menuentry "Reinstall via iPXE" {
          chainloader /ipxe.efi
        }
      '';
    };
  };
}

Presumably, this only works on UEFI systems.

Finding out how to create this entry wasn’t trivial at all, I couldn’t find anyone else doing it this way, but it seems to work perfectly fine.

Having this boot entry on its own isn’t much different from choosing a different one-time boot option, so I’ll look into using the grub-reboot tool to be able to trigger it remotely, although I haven’t tested that yet on NixOS.

2023/08/24 Update: I can confirm that grub-reboot works as expected on NixOS.

For example, you can get the available entries on your system like so:

[root@nixos:~]# grep menuentry /boot/grub/grub.cfg
menuentry "NixOS - Default" --class nixos {
menuentry "Windows 10" {
menuentry "iPXE Boot" {
menuentry "NixOS - Configuration 19 (2023-08-24 - 23.05.20230820.475d5ae)" --class nixos {

From this output, we’d run grub-reboot 2 to boot to iPXE on next reboot.

There’s also a more complicated approach, where the iPXE boot entry won’t be visible by default, only when you trigger it remotely.

To do this, we use the following snippet instead:

{ config, lib, pkgs, ... }:

{
  environment.shellAliases.reboot2PXE =
    "${pkgs.grub2}/bin/grub-editenv /boot/grub/grubenv set entry=ipxe && reboot";

  boot.loader = {
    efi.canTouchEfiVariables = true;
    grub = {
      enable = true;
      efiSupport = true;
      device = "nodev";

      extraFiles = { "ipxe.efi" = "${pkgs.ipxe}/ipxe.efi"; };
      extraConfig = ''
        if [ "''${entry}" = "ipxe" ]; then
          set entry=""
          save_env --file /grub/grubenv entry
          menuentry "Reinstall via iPXE" {
            chainloader /ipxe.efi
          }
        fi
      '';
    };
  };
}

Now, if we call reboot2PXE, the system will immediately reboot to iPXE, while not showing the iPXE boot entry on regular boots.

Closing remarks

As you can see by these snippets, setting up an iPXE server doesn’t have to be a chore.

As basically every computer built in the last 10 years has proper iPXE booting support, when managing tens of bare metal machines, setups like these drastically make the sysadmins’ lives easier.

Although I believe it would still be technically possible to remotely configure and recover these systems using lower level tools like AMT, I feel this would create a more complicated setup. I’m still looking to experiment with MeshCentral to explore these lower level AMT capibilities, and that may become a future blog post.

References

• https://github.com/danderson/netboot/tree/main/pixiecore
• https://nixos.wiki/wiki/Netboot

Installing “vanilla” NixOS on the Pi isn’t too difficult, just follow the installation steps on the wiki: download the SD card image, decompress it, flash the image to the SD card and you’re good to go.

However, it gets more complicated when you try to go off the beaten path, such as when trying to do full-disk encryption or using filesystems other than ext4.

After some reading and some trial and error, here are the steps I took to get NixOS with ZFS on the Pi, with UEFI booting.

Requirements

Make sure you’ve enabled support for USB booting on your Raspberry Pi. If yours is considerably older, you might need to update its EEPROM before you can enable support for USB booting. For more details, check the documentation.

Installation steps

I started by booting the Pi off of a USB stick with the generic SD card AArch64 image but it would work just as well by initially booting off of an SD card.

As my installation target was a SATA SSD, I now connected it to one of the USB 3.0 ports on the Pi. I find using the Raspberry Pi with an SSD makes it feel more like a real computer, as the SSD is faster, has more storage capacity and will most certainly be more durable than a flimsy SD card, especially now that I’ll also be using the Pi as a home server.

Partitioning

wipefs -a /dev/sda

parted -a optimal /dev/sda -- mklabel gpt
parted -a optimal /dev/sda -- mkpart ESP fat32 1MiB 513MiB
parted -a optimal /dev/sda -- set 1 esp on
parted -a optimal /dev/sda -- mkpart primary linux-swap 513MiB 8705MiB
parted -a optimal /dev/sda -- mkpart primary 8705MiB 100%

mkfs.fat -F 32 -n boot /dev/sda1

mkswap -L swap /dev/sda2
swapon /dev/sda2

zpool create -O mountpoint=none -O atime=off -o ashift=12 -O acltype=posixacl -O xattr=sa -O compression=zstd -O dnodesize=auto -O normalization=formD zroot /dev/sda3
zfs create -o refreservation=1G -o mountpoint=none zroot/reserved
zfs create zroot/root

mount -t zfs -o zfsutil zroot/root /mnt
mkdir -p /mnt/boot
mount /dev/sda1 /mnt/boot

UEFI setup

We’ll now setup UEFI on the Pi. We do this by placing the files from the Raspberry Pi 4 UEFI Firmware Images on the boot partition:

nix-shell -p wget unzip
cd /mnt/boot
wget https://github.com/pftf/RPi4/releases/download/v1.33/RPi4_UEFI_Firmware_v1.33.zip
unzip RPi4_UEFI_Firmware_v1.33.zip
rm README.md
rm RPi4_UEFI_Firmware_v1.33.zip

2022/11/04 Update: As a reader pointed out, you can also do this step by just copying the files on the installation media (assuming /dev/sda is your installation disk):

mkdir /firmware
mount /dev/sda1 /firmware
cp /firmware/* /mnt/boot

NixOS installation

As per usual, we let NixOS generate a config for us:

nixos-generate-config --root /mnt

As we’re using ZFS, we’ll need to enable ZFS support:

boot.supportedFilesystems = [ "zfs" ];
networking.hostId = "<8 random numbers>";

And also specify the zfsutil mount option for each of our datasets (in the /mnt/etc/nixos/hardware-configuration.nix file):

fileSystems."/" = {
  device = "zroot/root";
  fsType = "zfs";
  options = [ "zfsutil" ];
};

Now, as we’re booting with UEFI, we’ll delete the following line which is generated by default:

boot.loader.generic-extlinux-compatible.enable = true;

…and we’ll install a UEFI bootloader:

boot.loader.efi.canTouchEfiVariables = true;
boot.loader.systemd-boot.enable = true;

Finally, we’ll install NixOS:

nixos-install

Closing remarks

To my surprise, I found that by using UEFI I could more easily achieve non-standard setups with NixOS on the Pi. I had some attempts at trying to install NixOS with ZFS with the usual booting setup on the Pi but I could never seem to boot it after the installation.

I should also note that setting up encryption and remote unlocking is not very difficult on this setup, as it only requires a little bit more of ZFS configuration.

After setting this up, a friend of mine pointed me to the following links:

• Planning for a better NixOS on ARM
• NixOS on ARM/UEFI

It turns out that I accidentally ended up building an ARM system which is aligned with the NixOS ARM lead’s view for NixOS ARM systems in the future, which I find amusing.

References

• https://mgdm.net/weblog/nixos-on-raspberry-pi-4
• https://nixos.wiki/wiki/NixOS_on_ARM
• https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4
• https://elis.nu/blog/2019/08/encrypted-zfs-mirror-with-mirrored-boot-on-nixos/
• https://nixos.wiki/wiki/ZFS
• https://nixos.org/manual/nixos/stable/index.html#sec-installation