As I’ve now started hosting services at home, my VPS needs are only hosting a personal mailserver, my personal websites, a headscale server and to serve as a bastion server for some services I host at home that I want to be accessible from the outside world.

This means that I don’t need anything too powerful and can go with something cheaper. My final options were Hetzner Cloud CPX11, Scaleway Stardust, BuyVM Slice 1024 and the netcup VPS 200 G10s.

I ended up choosing the netcup VPS as it seemed to be the best value for the money.

If I weren’t hosting email, I’d strongly consider simply using Cloudflare Tunnel ou Tailscale Funnel for exposing some services to the public Internet and skip the VPS altogether.

The following post describes how I set up NixOS on this VPS.

Enabling UEFI boot

The first step was to enable UEFI booting on the server. Quite simple, just go to the VPS dashboard website > Settings > UEFI Settings > Activate UEFI Boot.

Booting the NixOS ISO

Thankfully, netcup allows its users to boot their servers with their ISOs of choice. If all you want is a standard NixOS setup, nixos-infect may be enough for your needs. However, I want to install ZFS with encryption which requires formatting the existing partitions. By being able to boot from the NixOS ISO, we skip the headache of installing NixOS through other exotic methods, such as installing from a rescue image or from running a kexec on the original deployed Linux distro (and that is a matter for another post).

This meant I had to download the NixOS ISO and upload it to their FTP server. The credentials for uploading custom images can be found in Media > DVD Drive > Login data to FTP.

However, after running the usual SFTP command, I got the following error:

❯ sftp -P <sftp-port> <username>@<sftp-hostname>
Unable to negotiate with <sftp-hostname> port <sftp-port>: no matching host key type found. Their offer: ssh-rsa,ssh-dss

So I explicitly allowed ssh-rsa and it worked as expected:

sftp -oHostKeyAlgorithms=+ssh-rsa -P <sftp-port> <username>@<sftp-hostname>
sftp> put Downloads/nixos-minimal-22.11.1347.0bf3109eeb6-x86_64-linux.iso cdrom/

Now, to boot from the uploaded ISO, we’ll go to Own DVDs and click on attach DVD:

And reboot the server.

Installing NixOS

Networking the live ISO

After booting to NixOS on the server, I found it had no network connectivity. So I connected through VNC through the control panel and am now in the NixOS live ISO.

We’ll start by manually setting up the network, according to the Network section of the control panel:

ip addr add <server-ip>/<prefix-length> dev <ethernet-network-interface>
ip route add default via <gateway-ip> dev <ethernet-network-interface>
echo "nameserver 1.1.1.1" > /etc/resolv.conf

We can now change the password for the root and finish the installation through SSH or just continue using the VNC window.

Partitioning

Now, we’ll wipe the drive and create its partitions:

wipefs -a /dev/sda

parted -a optimal /dev/sda
(parted) unit mib
(parted) mklabel gpt
(parted) mkpart ESP fat32 1 513
(parted) set 1 boot on
(parted) mkpart primary 513 100%
(parted) quit

We’ll now format the /boot partition:

mkfs.fat -F 32 -n boot /dev/sda1

Finally, we’ll create the ZFS pool for the persistent datasets:

ls /dev/disk/by-id/* # Find the correct disk.
ls /dev/disk/by-path/* # ... or if the VM is using virtio

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O acltype=posixacl \
    -O atime=off \
    -O canmount=off \
    -O compression=zstd \
    -O dnodesize=auto \
    -O normalization=formD \
    -O xattr=sa \
    -O mountpoint=none \
    -O encryption=on \
    -O keylocation=prompt \
    -O keyformat=passphrase \
    rpool /dev/disk/by-id/<disk>

zfs create -p -o refreservation=1G -o mountpoint=none rpool/local/reserved
zfs create -p rpool/local/nix
zfs create -p rpool/safe/persist

We’ll now start to mount our partitions and datasets so we can finally install NixOS:

mount -t tmpfs none /mnt
mkdir -p /mnt/{boot,nix,persist}

mount /dev/sda1 /mnt/boot
mount -t zfs -o zfsutil rpool/local/nix /mnt/nix
mount -t zfs -o zfsutil rpool/safe/persist /mnt/persist

Configuring NixOS

Let’s start by generating a configuration:

nixos-generate-config --root /mnt

And here follows some specific configuration, in addition to the one we’re used to.

Enable ZFS support:

/etc/nixos/configuration.nix

boot.supportedFilesystems = [ "zfs" ];
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
networking.hostId = <host-id>; # For example: head -c 8 /etc/machine-id

Set up the root user

/etc/nixos/configuration.nix

# To generate a hash to put in initialHashedPassword
# you can do this:
# $ nix-shell --run 'mkpasswd -m SHA-512 -s' -p mkpasswd
users.users.root.initialPassword = "hunter2";
users.mutableUsers = false;

You can also just skip this part and set up remote access by SSH. Be careful not to get locked out.

Partition and dataset configuration

/etc/nixos/hardware-configuration.nix

fileSystems."/" = {
    device = "none";
    fsType = "tmpfs";
    options = [ "defaults" "size=2G" "mode=755" ];
};

fileSystems."/boot" = {
    device = "/dev/disk/by-uuid/<boot_partition_uuid>";
    fsType = "vfat";
};

fileSystems."/nix" = {
    device = "rpool/local/nix";
    fsType = "zfs";
    options = [ "zfsutil" ];
};

fileSystems."/persist" = {
    device = "rpool/safe/persist";
    fsType = "zfs";
    options = [ "zfsutil" ];
    neededForBoot = true;
};

Swap

Enabling swap is usually a good idea. But we’d also like to avoid having swap on disk, as it’s really slow. So we’ll follow Fedora’s lead and set up swap on ZRAM: /etc/nixos/configuration.nix

zramSwap.enable = true;

Remote access through SSH

To enable remote SSH access with public key authentication, we’ll add the following:

/etc/nixos/configuration.nix

services.openssh = {
  enable = true;
  openFirewall = true;
  passwordAuthentication = false;
  kbdInteractiveAuthentication = false;
  hostKeys = [
    {
      bits = 4096;
      path = "/persist/etc/ssh/ssh_host_rsa_key";
      type = "rsa";
    }
    {
      path = "/persist/etc/ssh/ssh_host_ed25519_key";
      type = "ed25519";
    }
  ];
};

users.users.root.openssh.authorizedKeys.keys = [ "ssh_public_key" ];

Static IP configuration

Because netcup doesn’t provide DHCP, we’ll need to manually set up networking:

/etc/nixos/configuration.nix

networking = {
  networkmanager.enable = false;
  useDHCP = false;

  interfaces.ens3 = {
    useDHCP = false;

    ipv4.addresses = [{
      address = "<ipv4_address>";
      prefixLength = <ipv4_prefix_length>;
    }];

    ipv6.addresses = [{
      address = "<ipv6_address>";
      prefixLength = <ipv6_prefix_length>;
    }];
  };

  defaultGateway = "<ipv4_gateway>";
};

Persistence

Because we have root on tmpfs, we’ll need to persist specific files and folders we wish to keep. There are many ways to do this but I found using the impermanence module quite straightforward:

/etc/nixos/configuration.nix

environment.persistence."/persist" = {
  hideMounts = true;
  files = [
    "/etc/machine-id"
  ];

  directories = [
    "/var/log"
  ];
};

Remotely decrypting ZFS

We’ll now setup remote unlocking of the ZFS pool so we don’t need to VNC into the machine every time we reboot.

For this, we’ll need to enable networking in the initrd, which envolves enabling the correct kernel module. Not only that, if your provider doesn’t provide DHCP, you’ll also need to manually set up the static IPv4 address.

/etc/nixos/configuration.nix

boot = {
  # Set up static IPv4 address in the initrd.
  kernelParams = [ "ip=<ipv4_address>::<ipv4_gateway>:<ipv4_netmask>::<interface>:none" ];

  initrd = {
    # Switch this to your ethernet's kernel module.
    # You can check what module you're currently using by running: lspci -v
    kernelModules = [ "virtio_pci" ];

    network = {
      # This will use udhcp to get an ip address.
      # Make sure you have added the kernel module for your network driver to `boot.initrd.availableKernelModules`,
      # so your initrd can load it!
      # Static ip addresses might be configured using the ip argument in kernel command line:
      # https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt
      enable = true;
      ssh = {
        enable = true;
        # To prevent ssh clients from freaking out because a different host key is used,
        # a different port for ssh is useful (assuming the same host has also a regular sshd running)
        port = 2222;
        # hostKeys paths must be unquoted strings, otherwise you'll run into issues with boot.initrd.secrets
        # the keys are copied to initrd from the path specified; multiple keys can be set
        # you can generate any number of host keys using
        # `ssh-keygen -t ed25519 -N "" -f /path/to/ssh_host_ed25519_key`
        hostKeys = [ /path/to/ssh_host_ed25519_key_initrd ];
        # public ssh key used for login
        authorizedKeys = [ "<ssh_public_key>" ];
      };
      # this will automatically load the zfs password prompt on login
      # and kill the other prompt so boot can continue
      postCommands = ''
        cat <<EOF > /root/.profile
        if pgrep -x "zfs" > /dev/null
        then
          zfs load-key -a
          killall zfs
        else
          echo "zfs not running -- maybe the pool is taking some time to load for some unforseen reason."
        fi
        EOF
      '';
    };
  };
};

Then, after NixOS is installed, to decrypt the pool:

ssh -4 -p 2222 root@<ipv4_address>

Installing NixOS

We’re finally ready to install NixOS:

nixos-install --no-root-passwd --root /mnt
umount -Rl /mnt
zpool export -a

And don’t forget to detach the DVD before rebooting.

reboot

Closing remarks

Although this initial setup took some work, I’m happy with the final result.

By erasing our darlings, we keep our system clean and fresh. It’s also reassuring to know that everything that’s running is explicitly set in the NixOS configuration and that we can just move it to some other host. This also makes backups much easier, as I know exactly what needs to be kept.

I intend to have all my future servers set up like this. Hopefully, following this post will make it easier.

References

• https://elis.nu/blog/2019/08/encrypted-zfs-mirror-with-mirrored-boot-on-nixos/
• https://elis.nu/blog/2020/05/nixos-tmpfs-as-root
• https://nixos.wiki/wiki/ZFS