iPXE booting with NixOS

Table of Contents

When you only manage a few systems, it is usually enough to install NixOS on them using a live installer on a USB drive. When dealing with almost a hundred systems, however, using USB drives becomes unfeasable and solutions like PXE booting become necessary.

Thankfully, Pixiecore makes it easy to set up an iPXE server on NixOS, a task that could otherwise be quite complex.

The simplest iPXE server we can set up on NixOS can look like the following:

{ config, lib, pkgs, ... }:

{
  services.pixiecore = {
    enable = true;
    openFirewall = true;
    dhcpNoBind = true;
    kernel = "https://boot.netboot.xyz";
  };
}

In this configuration, we enable the Pixiecore service, set it to work along the DHCP server that already exists on our network and to boot the netboot.xyz iPXE image, a common option which gives you a selection of live installers on boot.

This may be enough for your environment but, as we’re running NixOS, we’d like to an image customized to our needs, having SSH access pre-configured, for example:

{ config, inputs, lib, pkgs, ... }:

let
  sys = inputs.nixos.lib.nixosSystem {
    system = "x86_64-linux";
    modules = [
      ({ config, pkgs, lib, modulesPath, ... }: {
        imports = [ (modulesPath + "/installer/netboot/netboot-minimal.nix") ];
        config = {
          services.openssh = {
            enable = true;
            openFirewall = true;

            settings = {
              PasswordAuthentication = false;
              KbdInteractiveAuthentication = false;
            };
          };

          users.users.root.openssh.authorizedKeys.keys = [
            "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI..."
          ];
        };
      })
    ];
  };

  build = sys.config.system.build;
in {
  services.pixiecore = {
    enable = true;
    openFirewall = true;
    dhcpNoBind = true; # Use existing DHCP server.

    mode = "boot";
    kernel = "${build.kernel}/bzImage";
    initrd = "${build.netbootRamdisk}/initrd";
    cmdLine = "init=${build.toplevel}/init loglevel=4";
    debug = true;
  };
}

The previous example does exactly that.

This now becomes very powerful and we’ve just starting using this on the lab computers we manage. We set the boot order to boot first from the hard disk and, if it’s not bootable, it will automatically boot from iPXE, still allowing remote access from the administrators and allowing us, if necessary, to redeploy its configuration with nixos-anywhere, as I’ll explain in a future blog post.

But we also want to be able to trigger an iPXE boot remotely, even if the current hard disk has a bootable configuration.

Initially, it wasn’t very obvious how we should be able to do this. One possibility would be to use efibootmgr to change the boot order accordingly, but it would make it more difficult to return to booting from local hard disk after the redeploy.

After some reflection, we settled on creating a special GRUB entry that boots from iPXE.

{ config, lib, pkgs, ... }:

{
  boot.loader = {
    efi.canTouchEfiVariables = true;
    grub = {
      enable = true;
      efiSupport = true;
      device = "nodev";

      extraFiles = { "ipxe.efi" = "${pkgs.ipxe}/ipxe.efi"; };
      extraEntries = ''
        menuentry "Reinstall via iPXE" {
          chainloader /ipxe.efi
        }
      '';
    };
  };
}

Presumably, this only works on UEFI systems.

Finding out how to create this entry wasn’t trivial at all, I couldn’t find anyone else doing it this way, but it seems to work perfectly fine.

Having this boot entry on its own isn’t much different from choosing a different one-time boot option, so I’ll look into using the grub-reboot tool to be able to trigger it remotely, although I haven’t tested that yet on NixOS.

2023/08/24 Update: I can confirm that grub-reboot works as expected on NixOS.

For example, you can get the available entries on your system like so:

[root@nixos:~]# grep menuentry /boot/grub/grub.cfg
menuentry "NixOS - Default" --class nixos {
menuentry "Windows 10" {
menuentry "iPXE Boot" {
menuentry "NixOS - Configuration 19 (2023-08-24 - 23.05.20230820.475d5ae)" --class nixos {

From this output, we’d run grub-reboot 2 to boot to iPXE on next reboot.

There’s also a more complicated approach, where the iPXE boot entry won’t be visible by default, only when you trigger it remotely.

To do this, we use the following snippet instead:

{ config, lib, pkgs, ... }:

{
  environment.shellAliases.reboot2PXE =
    "${pkgs.grub2}/bin/grub-editenv /boot/grub/grubenv set entry=ipxe && reboot";

  boot.loader = {
    efi.canTouchEfiVariables = true;
    grub = {
      enable = true;
      efiSupport = true;
      device = "nodev";

      extraFiles = { "ipxe.efi" = "${pkgs.ipxe}/ipxe.efi"; };
      extraConfig = ''
        if [ "''${entry}" = "ipxe" ]; then
          set entry=""
          save_env --file /grub/grubenv entry
          menuentry "Reinstall via iPXE" {
            chainloader /ipxe.efi
          }
        fi
      '';
    };
  };
}

Now, if we call reboot2PXE, the system will immediately reboot to iPXE, while not showing the iPXE boot entry on regular boots.

Closing remarks #

As you can see by these snippets, setting up an iPXE server doesn’t have to be a chore.

As basically every computer built in the last 10 years has proper iPXE booting support, when managing tens of bare metal machines, setups like these drastically make the sysadmins’ lives easier.

Although I believe it would still be technically possible to remotely configure and recover these systems using lower level tools like AMT, I feel this would create a more complicated setup. I’m still looking to experiment with MeshCentral to explore these lower level AMT capibilities, and that may become a future blog post.

Closing remarks #

References #