Self-Hosting on Carlos Vaz

Rootless Podman and Docker-Compose on NixOS

carlos@carjorvaz.com (Carlos Vaz) — Sun, 06 Aug 2023 00:00:00 +0000

At my university’s Computer Science and Engineering department, we want to allow students to use containers in the lab computers while not giving them root priviliges, as running Docker usually requires.

In the past, on our Ubuntu systems, we had a setup of rootless-Docker I’m not sure ever worked properly. There also seems to exist an option on NixOS to enable rootless-Docker, but we also had some issues in using it.

So we went with Podman, a Docker alternative that’s more ready to be run rootless by design. Also, Docker can just be aliased to Podman and the experience is so identical that the students may not even notice that they’re not running Docker proper.

Enabling Podman on NixOS is quite trivial but having it work in a rootless environment requires more configuration:

{ config, lib, pkgs, ... }:

{
  virtualisation = {
    containers.enable = true;
    containers.storage.settings = {
      storage = {
        driver = "overlay";
        runroot = "/run/containers/storage";
        graphroot = "/var/lib/containers/storage";
        rootless_storage_path = "/tmp/containers-$USER";
        options.overlay.mountopt = "nodev,metacopy=on";
      };
    };

    oci-containers.backend = "podman";
    podman = {
      enable = true;
      enableNvidia = true;
      dockerCompat = true;
      # extraPackages = [ pkgs.zfs ]; # Required if the host is running ZFS
    };
  };

  environment.systemPackages = with pkgs; [ docker-compose ];
  environment.extraInit = ''
    if [ -z "$DOCKER_HOST" -a -n "$XDG_RUNTIME_DIR" ]; then
      export DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock"
    fi
  '';
}

We set the DOCKER_HOST environment variable on user login so that docker-compose works using the rootless Podman socket.

Because students log into our lab computers using Kerberos, they don’t have /etc/subuid and /etc/subgid entries automatically created for them. So, in this particular setup, we also have a pam_exec hook to create the entries for them, otherwise Podman won’t work.

{
  # subidappend is a script we wrote that adds subuid and subgid entries on user login
  security.pam.services.login.text = lib.mkDefault (lib.mkAfter ''
    session optional pam_exec.so ${pkgs.subidappend}/bin/subidappend
  '');

  security.pam.services.sshd.text = lib.mkDefault (lib.mkAfter ''
    session optional pam_exec.so ${pkgs.subidappend}/bin/subidappend
  '');

  # Clean subuids and gids on boot
  systemd.tmpfiles.rules =
    [ "f+  /etc/subuid 0644 root root -" "f+  /etc/subgid 0644 root root -" ];
}

Closing remarks

We find this setup to be quite robust, it tends to just work with most Docker workflows, with the execption of using ports under 1024.

This allows students to run almost everything they wish to run on the lab computers, while still not having root privileges. The Nix package manager also helps with this. More importantly, students can now easily run services like Nginx and PostgreSQL, which are not trivial at all to run on the host system without root.

References

The Holy Grail Nextcloud setup made easy by NixOS

carlos@carjorvaz.com (Carlos Vaz) — Sat, 05 Aug 2023 00:00:00 +0000

Nextcloud really is the central piece to most people’s self-hosted infrastructure.

Providing file synchronization, a web interface to navigate through them, calendar, contacts, tasks, kanban and webmail, it presents itself as a complete GSuite self-hosted alternative.

Hosting Nextcloud has become easier over time, thanks to its docker-compose example setups and to the Snap for use mostly on Ubuntu systems. However, having a faster and more optimized setup can take some effort on these platforms. Thankfully, on NixOS it’s not hard at all, as I’ll show you.

{ self, config, lib, pkgs, ... }:

{
  services = {
    nginx.virtualHosts = {
      "cloud.example.com" = {
        forceSSL = true;
        enableACME = true;
      };

      "onlyoffice.example.com" = {
        forceSSL = true;
        enableACME = true;
      };
    };

    nextcloud = {
      enable = true;
      hostName = "cloud.example.com";

       # Need to manually increment with every major upgrade.
      package = pkgs.nextcloud27;

      # Let NixOS install and configure the database automatically.
      database.createLocally = true;

      # Let NixOS install and configure Redis caching automatically.
      configureRedis = true;

      # Increase the maximum file upload size to avoid problems uploading videos.
      maxUploadSize = "16G";
      https = true;
      enableBrokenCiphersForSSE = false;

      autoUpdateApps.enable = true;
      extraAppsEnable = true;
      extraApps = with config.services.nextcloud.package.packages.apps; {
        # List of apps we want to install and are already packaged in
        # https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/nextcloud/packages/nextcloud-apps.json
        inherit calendar contacts mail notes onlyoffice tasks;

        # Custom app installation example.
        cookbook = pkgs.fetchNextcloudApp rec {
          url =
            "https://github.com/nextcloud/cookbook/releases/download/v0.10.2/Cookbook-0.10.2.tar.gz";
          sha256 = "sha256-XgBwUr26qW6wvqhrnhhhhcN4wkI+eXDHnNSm1HDbP6M=";
        };
      };

      config = {
        overwriteProtocol = "https";
        defaultPhoneRegion = "PT";
        dbtype = "pgsql";
        adminuser = "admin";
        adminpassFile = "/path/to/nextcloud-admin-pass";
      };
    };

    onlyoffice = {
      enable = true;
      hostname = "onlyoffice.example.com";
    };
  };
}

You may want to proceed with caution while setting up the OnlyOffice server, which will allow for Google Docs-like functionality on our Nextcloud instance, by having it only accessible inside your VPN or by setting the services.onlyoffice.jwtSecretFile option if exposed to the public Internet.

With this snippet, a Nextcloud instance with a selection of pre-installed Apps, PostgreSQL as a database, Redis Caching and Let’s Encrypt certificates will be set up for you.

To connect to the OnlyOffice server, configure it appropriately in Administration settings > ONLYOFFICE > ONLYOFFICE Docs address.

Backups

In this configuration, we need to persist the /var/lib/nextcloud and /var/lib/postgresql directories.

For backing up, you could copy /var/lib/nextcloud to another computer and, for the database, dump it to a file and copy it to another computer as well, as described in the official Nextcloud documentation.

Conclusion

Once again, NixOS proves itself as an amazing self-hosting platform.

Nextcloud, in its default configuration, is sometimes known for running slow. Thanks to NixOS, we’ve optimized its performance and that’s quite impactful, as it’s my most used self-hosted application. Having all of these apps running on Nextcloud has enabled me to move on from GSuite to a mostly autonomous and self-hosted infrastructure.

In the future, I look forward to being able to use Collabora/Nextcloud Office instead of OnlyOffice, as it’s more aligned with Nextcloud’s philosophical goals and hasn’t done suspicious decisions in the past.

References

Setting up Plausible Analytics on NixOS

carlos@carjorvaz.com (Carlos Vaz) — Thu, 03 Aug 2023 00:00:00 +0000

I’ve been looking to set up analytics on my blog and maybe some other websites. I’ve avoided Google Analytics as it’s quite insideous and would like to avoid giving them my readers’ data willingly and for free.

From my research, I found Plausible and Umami to be good alternatives. Both collect so little data that cookie consent banners are not required and both are very minimalistic in their dashboards, focusing on what really matters.

A friend of mine is using Umami and is quite happy with his setup. Umami should also be lighter, as Plausible is designed to support a much larger number of connected websites. Despite all of that, I ended up choosing Plausible simply because it has an upstream NixOS module, allowing me to set it up quite trivially, whereas I would have to package Umami and setup its PostgreSQL integration manually or use containers, which I’m also trying to avoid in my self-hosted infrastructure.

This simple snippet is enough to set up a Plausible instance, all automatically configured for you.

{ self, config, lib, pkgs, ... }:

let domain = "plausible.example.com";
in {
  age.secrets = {
    plausibleAdminPassword.file = "${self}/secrets/plausibleAdminPassword.age";
    plausibleReleaseCookie.file = "${self}/secrets/plausibleReleaseCookie.age";
    plausibleSecretKeybase.file = "${self}/secrets/plausibleSecretKeybase.age";
  };

  services = {
    nginx.virtualHosts.${domain} = {
      forceSSL = true;
      enableACME = true;
      locations."/".proxyPass =
        "http://127.0.0.1:${toString config.services.plausible.server.port}";
    };

    plausible = {
      enable = true;

      adminUser = {
        # activate is used to skip the email verification of the admin-user that's
        # automatically created by plausible. This is only supported if
        # postgresql is configured by the module. This is done by default, but
        # can be turned off with services.plausible.database.postgres.setup.
        activate = true;
        email = "john.doe@example.com";
        passwordFile = config.age.secrets.plausibleAdminPassword.path;
      };

      server = {
        baseUrl = "https://${domain}";
        secretKeybaseFile = config.age.secrets.plausibleSecretKeybase.path;
      };
    };
  };

  # As this is a root on tmpfs system, we use the impermanence
  # NixOS module to persist state between reboots.
  # You can omit the next block if using a regular configuration.
  environment.persistence."/persist".directories =
    [
      "/var/lib/postgresql"
      {
        directory = "/var/lib/clickhouse";
        mode = "0750";
        user = "clickhouse";
        group = "clickhouse";
      }
      {
        directory = "/var/lib/private/plausible";
        mode = "0750";
        user = "plausible";
        group = "plausible";
      }
    ];
}

In this snippet I’m using agenix to manage secrets such as the admin password. You’re obvisouly free to use your secrets management tool of choice or you can just use file paths which are not in public version control systems.

As per the official documentation, the secretKeybaseFile’s contents should be generated with:

$ openssl rand -base64 64 | tr -d '\n' ; echo

For some unknown unreason, the admin password I set up didn’t work after the rebuild. I worked around that by using the reset password form on the login page. In my case, this worked out of the box as the host where I deployed this Plausible instance was also a mailserver. If yours isn’t, you may have to adjust the services.plausible.mail.* options accordingly.

All that’s left is to add a website in the dashboard. You’ll be presented with an HTML snippet which you should include on your website’s tag to start collecting privacy-respecting analytics.

As optional configuration, I enabled email alerts for weekly and monthly reports and also traffic spikes, to avoid creating the common addiction of being permanently looking at the analytics dashboard instead of engaging in productive activities.

Conclusion

I’m quite happy with this setup. As I said before, this gives me some insight into my readership without compromising their privacy.

For many months my only analytics were from the Google Search Console, as I was having configuration and compilation errros in the past when trying to set up Plausible in NixOS. Recently I decided to give it another go for no special reason and was pleasantly surprised to find out that it just works.

It seems to be possible to integrate Plausible with the Google Search Console API and that may be something I’ll look into in the future.

If you wish to work around your readers’ adblockers, this also seems to be possible using a proxy as to not trigger the usual adblocker filters, as explained in this article.

References

Setting up a WordPress website on NixOS

carlos@carjorvaz.com (Carlos Vaz) — Tue, 01 Aug 2023 00:00:00 +0000

Another instance where NixOS makes it easy to self-host services, this time we’re setting up WordPress using the upstream nixpkgs module and some auxiliary functions to help us package themes and plugins.

{ config, lib, pkgs, ... }:

let
  domain = "example.com";

  # Auxiliary functions
  fetchPackage = { name, version, hash, isTheme }:
    pkgs.stdenv.mkDerivation rec {
      inherit name version hash;
      src = let type = if isTheme then "theme" else "plugin";
      in pkgs.fetchzip {
        inherit name version hash;
        url = "https://downloads.wordpress.org/${type}/${name}.${version}.zip";
      };
      installPhase = "mkdir -p $out; cp -R * $out/";
    };

  fetchPlugin = { name, version, hash }:
    (fetchPackage {
      name = name;
      version = version;
      hash = hash;
      isTheme = false;
    });

  fetchTheme = { name, version, hash }:
    (fetchPackage {
      name = name;
      version = version;
      hash = hash;
      isTheme = true;
    });

  # Plugins
  google-site-kit = (fetchPlugin {
    name = "google-site-kit";
    version = "1.103.0";
    hash = "sha256-8QZ4XTCKVdIVtbTV7Ka4HVMiUGkBYkxsw8ctWDV8gxs=";
  });

  # Themes
  astra = (fetchTheme {
    name = "astra";
    version = "4.1.5";
    hash = "sha256-X3Jv2kn0FCCOPgrID0ZU8CuSjm/Ia/d+om/ShP5IBgA=";
  });

in {
  services = {
    nginx.virtualHosts.${domain} = {
      enableACME = true;
      forceSSL = true;
    };

    wordpress = {
      webserver = "nginx";
      sites."${domain}" = {
        plugins = { inherit google-site-kit; };
        themes = { inherit astra; };
        settings = { WP_DEFAULT_THEME = "astra"; };
      };
    };
  };

  # As this is a root on tmpfs system, we use the impermanence
  # NixOS module to persist WordPress state between reboots.
  # You can omit the next two lines if using a regular configuration.
  environment.persistence."/persist".directories =
    [ "/var/lib/mysql" "/var/lib/wordpress" ];
}

I configure WordPress to use Nginx as the web server instead of httpd as it’s what I already use as a reverse proxy for my other self-hosted applications.

Setting up Headscale on NixOS

carlos@carjorvaz.com (Carlos Vaz) — Sun, 11 Dec 2022 00:00:00 +0000

I’ve recently bought a used USFF computer at a good price, inspired by the TinyMiniMicro project. It’s super fast, has plenty of storage and memory (32GB of RAM and 2TB of NVMe SSD storage) and, more importantly, is almost completely silent and consumes almost no electricity (~12W in idle, about 2x Raspberry Pi 4).

So I decided to start moving my self-hosted applications to this machine, which will become my home server. I will begin using my Kimsufi dedicated server only as an email server and as my only system open to the public Internet, as it has a static public IP address and OVH will know how to deal with DDOS attacks and such better than me. Even though the Kimsufi machine is at an unbeatable price, around 7€ per month for 2TB of storage, its processor is incredibly slow and under-powered, while also having its networking capped at 100MBps.

This new home server will be faster while also being more in line with the philosophy of self-hosting, to be the owner of my hardware. Not only that, it will also have better networking, not only speed-wise but also because it’ll be geographically closer to me.

Headscale

But I don’t want to have applications running at home being exposed to the public internet, where I would use DynamicDNS and port-forwarding on my router. The Internet can be very hostile and chaotic, so I’ll want my services to only be accessible to people I know and trust. Well, this is the pitch for using a mesh-VPN, where we build a private network where devices can communicate with each other wherever they are, but are inaccessible from the rest of the Internet.

I have already used Nebula in the past as a mesh-VPN solution. It’s a self-hosted solution, is well integrated with NixOS and has an Android client. However, after the initial setup, I felt too much friction to add new devices to the network. I would need to generate keys for each device and then transfer them to my laptop, where I had my CA, sign the keys and then transfer them back.

All over the Web, there has been much talk about this new product called Tailscale. It was also designed with this usecase in mind and many have described as a new Hamachi, which I have used in the past for having local Minecraft servers for playing with friends.

Although many of Tailscale’s components are open-source, especially its clients, the server is closed-source. It’s more or less understandable that it works this way. They provide Tailscale as a SaaS product, especially directed at the enterprise sector, and this allows them to keep developing and innovating. If they made it all open-source, they would risk ending up in a Docker situation, where they would struggle to monetize their oferring.

However, I was looking for free and open-source solution, hosted by myself. I then found out that there’s an open-source implementation of the Tailscale server, called Headscale (perfect naming). Sometimes, it evens gets contributions from Tailscale employees. One of its most important maintainers currently works at Tailscale.

Excellent, Headscale is also well integrated with NixOS and I could basically have all the advantages of using Tailscale such as MagicDNS, Taildrop, Exit Nodes, etc. while only using open-source components and a server managed by myself.

Installing Headscale on NixOS

I couldn’t find any post showing me how to setup a Headscale server on NixOS. I only managed to find another person’s configuration. Turns out, it wasn’t that hard. All that was needed was to activate the Headscale service as set the relevant options:

let domain = "headscale.example.com";
in {
  services = {
    headscale = {
      enable = true;
      address = "0.0.0.0";
      port = 8080;
      server_url = "https://${domain}";
      dns = { baseDomain = "example.com"; };
      settings = { logtail.enabled = false; };
    };

    nginx.virtualHosts.${domain} = {
      forceSSL = true;
      enableACME = true;
      locations."/" = {
        proxyPass =
          "http://localhost:${toString config.services.headscale.port}";
        proxyWebsockets = true;
      };
    };
  };

  environment.systemPackages = [ config.services.headscale.package ];

2024/04/27 Update: A reader pointed out that the option serverUrl is now server_url, and that the error wasn’t too obvious. It’s now updated above.

And it just worked. I should note that the option proxyWebsockets is need according to the docs.

All that’s left is to create our first namespace, to which we’ll our nodes later on:

headscale namespaces create

Using the Tailscale client and daemon on NixOS

Using the Tailscale client on NixOS is also really easy. It’s just a matter of activating the following configuration:

services.tailscale.enable = true;
networking.firewall = {
  checkReversePath = "loose";
  trustedInterfaces = [ "tailscale0" ];
  allowedUDPPorts = [ config.services.tailscale.port ];
};

And running the command to add the node to the network:

tailscale up --login-server

This command will output a link with instructions to add the node to the specified namespace on the server-side, which will look something like:

headscale --namespace  nodes register --key

Where is the string at the end of the URL shown by the Tailscale client.

After that, I noticed that, by default, the node names end up being the hostname followed by a string of random characters. In order to make the names more legible and memorable, I chose to change them to be the machines’ hostname only. I have read that using the option --hostname when running the tailscale up command sets the hostname right away but I haven’t tested it yet.

To rename a single node on headscale:

headscale nodes list
headscale node rename -i

MagicDNS

As I use systemd-resolved on my systems, Tailscale takes care of setting up all of the DNS-related configuration. However, if I didn’t use systemd-resolved, I’d have to configure the DNS myself, as explained here.

Thanks to MagicDNS, after setting up the clients, we can connect to any node in the network, wherever we are in the world, by referring to them as ... For example:

ping node1.headnet.example.com

Using the Tailscale Android client on GrapheneOS

Setting up Tailscale on Android is also supposed to be quite straightforward, especially because Tailscale added a way to use external servers, allowing Headscale users to use the same client (link).

However, every time I tapped the “Save and Restart” button, the app would crash immediately. When I restarted it, the sign-in button kept redirecting me to Tailscale’s login page, which led me to believe it didn’t actually configure the app to use my Headscale server.

Another user seemed to have just the same problem (link), but there were no answers.

My remaining option was to modify the client’s source code in order to use my Headscale server by default. It was a quite simple modification (link) but it requires me to sync my own fork and build the app every time there’s an update.

To make matters worse, Tailscale’s client is also incompatible with the Private DNS setting on Android, which I used in order to have DNS-level ad-blocking, using Adguard. I’ll have to investigate further if it makes sense to set up a DNS server on my home server and use it on my Android device so I can have ad-blocking again (according to Daniel Micay, this is the only reasonable way to have system-wide ad-blocking on an Android device).

Closing remarks

The overall feeling of this experiment was that setting all of this up was very straightforward, with the exception of the Android client on my phone. Admittedly, it’s also a more esoteric setup than usual (GrapheneOS).

In contrast to Nebula, adding new devices to the network is a matter of running a command both on the client and on the server (or tapping a button if on an Android device) and not of sending files around.

Also, MagicDNS and Taildrop just work and are great additions.

All that’s left is to set up services on my home-server, in order for this mesh-VPN be of any use.

It would be interesing to keep having Let’s Encrypt SSL certificates on my services, running inside the private network. However, this will be harder than usual as I won’t be able to complete the HTTP-01 challenge. I’ll need to setup the DNS-01 challenge which will allow me to use wildcard certificates. But that’s a matter for a future post.