Setting up Headscale on NixOS

I’ve recently bought a used USFF computer at a good price, inspired by the TinyMiniMicro project. It’s super fast, has plenty of storage and memory (32GB of RAM and 2TB of NVMe SSD storage) and, more importantly, is almost completely silent and consumes almost no electricity (~12W in idle, about 2x Raspberry Pi 4).

So I decided to start moving my self-hosted applications to this machine, which will become my home server. I will begin using my Kimsufi dedicated server only as an email server and as my only system open to the public Internet, as it has a static public IP address and OVH will know how to deal with DDOS attacks and such better than me. Even though the Kimsufi machine is at an unbeatable price, around 7€ per month for 2TB of storage, its processor is incredibly slow and under-powered, while also having its networking capped at 100MBps.

This new home server will be faster while also being more in line with the philosophy of self-hosting, to be the owner of my hardware. Not only that, it will also have better networking, not only speed-wise but also because it’ll be geographically closer to me.

Headscale #

But I don’t want to have applications running at home being exposed to the public internet, where I would use DynamicDNS and port-forwarding on my router. The Internet can be very hostile and chaotic, so I’ll want my services to only be accessible to people I know and trust. Well, this is the pitch for using a mesh-VPN, where we build a private network where devices can communicate with each other wherever they are, but are inaccessible from the rest of the Internet.

I have already used Nebula in the past as a mesh-VPN solution. It’s a self-hosted solution, is well integrated with NixOS and has an Android client. However, after the initial setup, I felt too much friction to add new devices to the network. I would need to generate keys for each device and then transfer them to my laptop, where I had my CA, sign the keys and then transfer them back.

All over the Web, there has been much talk about this new product called Tailscale. It was also designed with this usecase in mind and many have described as a new Hamachi, which I have used in the past for having local Minecraft servers for playing with friends.

Although many of Tailscale’s components are open-source, especially its clients, the server is closed-source. It’s more or less understandable that it works this way. They provide Tailscale as a SaaS product, especially directed at the enterprise sector, and this allows them to keep developing and innovating. If they made it all open-source, they would risk ending up in a Docker situation, where they would struggle to monetize their oferring.

However, I was looking for free and open-source solution, hosted by myself. I then found out that there’s an open-source implementation of the Tailscale server, called Headscale (perfect naming). Sometimes, it evens gets contributions from Tailscale employees. One of its most important maintainers currently works at Tailscale.

Excellent, Headscale is also well integrated with NixOS and I could basically have all the advantages of using Tailscale such as MagicDNS, Taildrop, Exit Nodes, etc. while only using open-source components and a server managed by myself.

Installing Headscale on NixOS #

I couldn’t find any post showing me how to setup a Headscale server on NixOS. I only managed to find another person’s configuration. Turns out, it wasn’t that hard. All that was needed was to activate the Headscale service as set the relevant options:

let domain = "headscale.example.com";
in {
  services = {
    headscale = {
      enable = true;
      address = "0.0.0.0";
      port = 8080;
      server_url = "https://${domain}";
      dns = { baseDomain = "example.com"; };
      settings = { logtail.enabled = false; };
    };

    nginx.virtualHosts.${domain} = {
      forceSSL = true;
      enableACME = true;
      locations."/" = {
        proxyPass =
          "http://localhost:${toString config.services.headscale.port}";
        proxyWebsockets = true;
      };
    };
  };

  environment.systemPackages = [ config.services.headscale.package ];

2024/04/27 Update: A reader pointed out that the option serverUrl is now server_url, and that the error wasn’t too obvious. It’s now updated above.

And it just worked. I should note that the option proxyWebsockets is need according to the docs.

All that’s left is to create our first namespace, to which we’ll our nodes later on:

headscale namespaces create <namespace_name>

Using the Tailscale client and daemon on NixOS #

Using the Tailscale client on NixOS is also really easy. It’s just a matter of activating the following configuration:

  services.tailscale.enable = true;
  networking.firewall = {
    checkReversePath = "loose";
    trustedInterfaces = [ "tailscale0" ];
    allowedUDPPorts = [ config.services.tailscale.port ];
  };

And running the command to add the node to the network:

tailscale up --login-server <headscale_url>

This command will output a link with instructions to add the node to the specified namespace on the server-side, which will look something like:

headscale --namespace <namespace_name> nodes register --key <machine_key>

Where <machine_key> is the string at the end of the URL shown by the Tailscale client.

After that, I noticed that, by default, the node names end up being the hostname followed by a string of random characters. In order to make the names more legible and memorable, I chose to change them to be the machines’ hostname only. I have read that using the option --hostname when running the tailscale up command sets the hostname right away but I haven’t tested it yet.

To rename a single node on headscale:

headscale nodes list
headscale node rename -i <node_id> <new_name>

MagicDNS #

As I use systemd-resolved on my systems, Tailscale takes care of setting up all of the DNS-related configuration. However, if I didn’t use systemd-resolved, I’d have to configure the DNS myself, as explained here.

Thanks to MagicDNS, after setting up the clients, we can connect to any node in the network, wherever we are in the world, by referring to them as <name>.<namespace>.<baseDomain>. For example:

ping node1.headnet.example.com

Using the Tailscale Android client on GrapheneOS #

Setting up Tailscale on Android is also supposed to be quite straightforward, especially because Tailscale added a way to use external servers, allowing Headscale users to use the same client (link).

However, every time I tapped the “Save and Restart” button, the app would crash immediately. When I restarted it, the sign-in button kept redirecting me to Tailscale’s login page, which led me to believe it didn’t actually configure the app to use my Headscale server.

Another user seemed to have just the same problem (link), but there were no answers.

My remaining option was to modify the client’s source code in order to use my Headscale server by default. It was a quite simple modification (link) but it requires me to sync my own fork and build the app every time there’s an update.

To make matters worse, Tailscale’s client is also incompatible with the Private DNS setting on Android, which I used in order to have DNS-level ad-blocking, using Adguard. I’ll have to investigate further if it makes sense to set up a DNS server on my home server and use it on my Android device so I can have ad-blocking again (according to Daniel Micay, this is the only reasonable way to have system-wide ad-blocking on an Android device).

Closing remarks #

The overall feeling of this experiment was that setting all of this up was very straightforward, with the exception of the Android client on my phone. Admittedly, it’s also a more esoteric setup than usual (GrapheneOS).

In contrast to Neubla, adding new devices to the network is a matter of running a command both on the client and on the server (or tapping a button if on an Android device) and not of sending files around.

Also, MagicDNS and Taildrop just work and are great additions.

All that’s left is to set up services on my home-server, in order for this mesh-VPN be of any use.

It would be interesing to keep having Let’s Encrypt SSL certificates on my services, running inside the private network. However, this will be harder than usual as I won’t be able to complete the HTTP-01 challenge. I’ll need to setup the DNS-01 challenge which will allow me to use wildcard certificates. But that’s a matter for a future post.