My experience of managing a cloud-hosted server with NixOS

It’s been some time since I’ve used NixOS as a desktop some time ago. But lots of things happened since then.

Long story short, I stopped using NixOS after the previously linked post and reverted to using a traditional Linux system. I did keep using Nix package manager to easily create reproducible development environment for projects.

However, I’ve gone back to using it sometime at the start of 2022 and used it throughout the year. At the start of this year, however, I’m starting to learn managing Linux-based systems for servers. What could be a better way (not necessarily as we’ll discuss it) of managing it with the devil I know: NixOS.

In this post, I’ll be journaling my experience managing a NixOS server for 2 months. [1]

Why NixOS? (redux version)

As you may already heard from any staunch advocates which I’ve even said something about those lines myself, there are a couple of good reasons to use NixOS. But I’m fairly sure you’ve heard a fair share of the same bullet points at this point so let me give you my perspective from using it for some time instead.

What I like about it is the perspective of building NixOS systems just like how one would program applications. You create a text file containing a Nix program, you pass the program to the compiler (or in this case, the Nix build daemon), then you "compiled" an operating system.

Furthermore, the overall Nix ecosystem has some things helping to create a NixOS system such as nixpkgs which does not only have one of the largest package repositories (as of 2023-02-14) but also a wide set of options including…​

  • Services from server-oriented programs (e.g., nginx, Bind, borgbackup) to complete desktop sessions (e.g., GNOME, KDE Plasma, i3, bspwm) with additional options to configure them.

  • Various programs such as tmux, vim, and command-line shells (e.g., Bash, zsh, fish) which also includes configuration options for them.

  • Hardware tweaks such as installing certain hardware drivers, a declarative networking setup (which is very useful for servers), and setting up filesystems complete with mount options.

All of those can be made to create NixOS systems for a variety of purposes.

Prior to this post, I use NixOS as my desktop OS of choice for at least a year. Let me show you some examples of how I use it.

The point is…​ NixOS is very flexible despite that it already gives you a complete set of knobs to tinker with especially with mechanisms in place such as Nix modules [3], nixpkgs overrides, overlays, and flakes.

Unlike configuring from a traditional Linux system (e.g., Debian, Arch, Fedora), NixOS is nicer in terms of UX for configuring the system since you only have to monitor one location to consolidate the system configuration.

The Nix ecosystem

It’s not as large as other ecosystems such as from Docker with its vast selection of images and widespread tools such as Kubernetes. The amount of things within the containers ecosystem is just too huge which rallies even more support behind it.

This doesn’t mean the Nix ecosystem is something to be scoffed at. There’s more to what you can do with NixOS especially there’s an ecosystem beyond nixpkgs.

Here’s some of the tools I use related to managing the server.

  • home-manager is quite similar to NixOS that it instead of building an operating system, it builds a home environment allowing you to specify user-specific tweaks that is otherwise not available from nixpkgs that are more focused on the holistic side of operating systems (i.e., system-wide services, programs, and configurations).

    While most of the options found here are oriented toward desktop usage, it’s still useful for servers (e.g., putting certain files in its home directory).

  • nixpkgs can build NixOS systems into various formats including container images, a plethora of cloud provider-specific image formats, and virtual machines. nixos-generators builds upon this as a nice frontend to easily make use of it.

    I’ve used it to generate a personalized NixOS installer as part of a release from my config. At some point, I’ve also used it to generate a Google Cloud image that can easily create a Compute instance from it.

  • deploy-rs is my deployment tool of choice. It comes with niceties such as magic rollback which is useful if it rendered your configuration to be unconnectable. Not to mention, it can deploy other than NixOS systems such as home-manager configurations [4].

  • Putting sensitive credentials is one of the weaker points in a NixOS system especially that the Nix store directory is readable by anyone. At this point, there’s a bunch of secret management tools integrated with NixOS with various way of getting around that fact. My tool of choice is sops-nix given that sops is simpler (compared to other tools) and can integrate with several key formats [5].

Configuring the server

I’m managing one instance in Hetzner Cloud. While it doesn’t have a NixOS image, it is easy to initiate a NixOS instance with the nixos-infect script which will…​ infect a Linux system into a NixOS one (without the spreading of NixOS disease though).

Thankfully, this can be automated since Hetzner also has an ecosystem of libraries and tools for more flexibility including a command-line interface for managing Hetzner Cloud deployments.

Here’s a script using hcloud to easily create one such setup
hcloud network create --name smn-lan --ip-range 172.16.0.0/12
hcloud server create --location hel1 --type cx21 --image ubuntu-22.04 \
    --network smn-lan \
    --user-data-from-file ./hosts/plover/files/hcloud/hcloud-user-data.yml \
    --ssh-key foodogsquared@foodogsquared.one \
    --name nixos-plover

Looking into NixOS as a server is pretty similar to how one would configure any Linux-based systems as a server.

If you’re looking into managing a NixOS system in Hetzner Cloud, I recommend to not start with executing the nixos-infect script and explore the system a little bit. This is important to see the network interfaces configuration and configure your networking interfaces based from it.

One would have to statically configure the hardware of the server, mostly relating to networking setup. For this, I used systemd-networkd service that integrates well in NixOS.

The following systemd-networkd configuration is based from the generated systemd network unit from Ubuntu 22.04 image from Hetzner Cloud. It assumes that you have an global IPv4 and IPv6 address and a private IPv4 subnet from networks feature.

{ config, options, lib, pkgs, ... }:

let
  # IPv4 is already configured through DHCP so no need to configure it.
  wan = {
    IPv6.address = "2001:db8:4723:d1ad::";
    IPv6.gateway = "fe80::1";
  };

  lan = {
    IPv4.address = "10.32.54.1";
    IPv4.gateway = "10.0.0.1";
    IPv6.address = "fd45:5643:1ade::";
    IPv6.gateway = "fe80::1";
  };
in
{
  systemd.network.networks = {
    "10-wan" = with wan; {
      matchConfig.Name = [ "ens3" "enp0s3" ];

      # Setting up IPv6.
      address = [ "${IPv6.address}/64" ];
      gateway = [ IPv6.gateway ];

      networkConfig = {
        # IPv6 has to be manually configured.
        DHCP = "ipv4";

        LinkLocalAddressing = "ipv6";
        IPForward = true;

        # This is based from
        # https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.
        DNS = [
          "2a01:4ff:ff00::add:2"
          "2a01:4ff:ff00::add:1"
        ];
      };
    };

    "20-lan" = with lan; {
      matchConfig.Name = [ "ens10" "enp0s10" ];

      address = [
        "${IPv4.address}/16"
        "${IPv6.address}/64"
      ];

      gateway = [
        IPv4.gateway
        IPv6.gateway
      ];
    };
  };
}
Networking configuration in NixOS

Currently (as of NixOS 23.05-unstable), there are two main ways to declaratively configure the networking setup: networking.interfaces and systemd.network.

networking.interfaces is considered the default seeing as it is used on the installers for NixOS. Although, there seems to be interest to switch to systemd-networkd as the default way — not to mention, part of the description of the networking.interfaces recommends using systemd.network.netdevs instead.

If this would be configured with networking.interfaces, then it would look something like the following listing (without the other aspects of networking such as proper routing).

{ config, options, lib, pkgs, ... }:

let
  # IPv4 is already configured through DHCP so no need to configure it.
  wan = {
    IPv6.address = "2001:db8:4723:d1ad::";
    IPv6.gateway = "fe80::1";
  };

  lan = {
    IPv4.address = "10.32.54.1";
    IPv4.gateway = "10.0.0.1";
    IPv6.address = "fd45:5643:1ade::";
    IPv6.gateway = "fe80::1";
  };
in
{
  networking = {
    enableIPv6 = true;
    defaultGatewayIPv6 = wan.IPv6.gateway;

    interfaces = {
      enp0s3 = {
        useDHCP = true;
        ipv6.addresses = [
          {
            address = wan.IPv6.address;
            prefixLength = 64;
          }
        ];
      };

      enp0s10 = {
        ipv4.addresses = [
          {
            address = lan.IPv4.address;
            prefixLength = 16;
          }
        ];

        ipv6.addresses = [
          {
            address = lan.IPv6.address;
            prefixLength = 64;
          }
        ];
      };
    };
  };
}

In a typical server setup, one would harden the server for security. Otherwise, unwanted traffic including potential attackers might take advantage of it. Fortunately, nixpkgs already has a profile that already does most of that for you and all you have to do is to import it in your NixOS configuration like in the following code listing.

{ config, modulesPath, ... }:

{
  imports = [ "${modulesPath}/profiles/hardened.nix" ];
}

With the most important things done, you can now proceed with the service configuration. Here’s a non-exhaustive list of things that I’ve done for the server.

  • systemd-networkd can configure Wireguard interfaces since v237 so I eventually used it for configuring Wireguard tunnels instead of standing up an OpenVPN server.

  • One could create a non-root user dedicated for managing the system. This could also be used to avoid using root directly, only using sudo when needed to while inside of the system.

    In my case, I created two non-root users: specifically to be used for deployment and one to be used as the entry point for the server since the former doesn’t have a password to enter privilege mode. The latter has a customized user environment done through home-manager.

  • Enable all of the services you want which I’ve included some details in the next chapter. Of course, as long as it fits in the server with its expected amount of traffic and the average amount of resources.

  • Setting a remote backup service. In this case, I’ve chose to use BorgBackup as it is also what I’ve previously used anyways. Not to mention, it is pretty nice what it does out-of-the-box compared to other services.

    I’ve set the backup service with a Hetzner Storage Box which is surprisingly cheap at 1TB for 4€ a month. All you have to do at that point is to enable SSH to add the Borg server then add the SSH keys.

The results and the maintenance process

The resulting NixOS server configuration is all available in my NixOS configuration repo. At the end, I was able to deploy instances of some of the applications I want to host and manage myself (among other things).

Alongside some important services to keep them in check such as the firewall for basic measures, fail2ban to slow down intrusions from these publicly available services, and using a hardened kernel.

I really like how it turned out. Though I’m worried how I’m putting it all in one basket but I think it’s mostly fine for a smaller instance as long as it is hardened and take precautions properly.

Extending and modifying NixOS services

As previously mentioned, NixOS is very flexible while giving you finer controls for configuring several aspects of the configuration especially for system services.

In the following example, I modified the CoreDNS service to add service credentials to enable DNS-over-TLS using the generated certificate from the ACME client. I also made one more change to modify the DNS zone file with sensitive information before the service is activated.

The modified CoreDNS service with DNS-over-TLS and insert sensitive information
{ config, lib, pkgs, ... }:

let
  inherit (config.networking) domain;
  domainZoneFile = ./config/coredns/${domain}.zone;
  domainZoneFile' = "/etc/coredns/zones/${domain}.zone";

  dnsDomainName = "ns1.${domain}";
  corednsServiceName = "coredns";
in
{
  # Generating a certificate for the DNS-over-TLS feature.
  security.acme.certs."${dnsDomainName}".postRun = ''
    systemctl restart ${corednsServiceName}.service
  '';

  services.coredns.config = ''
    tls://. {
      tls {$CREDENTIALS_DIRECTORY}/cert.pem {$CREDENTIALS_DIRECTORY}/key.pem {$CREDENTIALS_DIRECTORY}/fullchain.pem
      forward . /etc/resolv.conf
    }
  '';

  systemd.services.${corednsServiceName} = {
    requires = [ "acme-finished-${dnsDomainName}.target" ];
    preStart =
      let
        secretsPath = path: config.sops.secrets."plover/${path}".path;
        replaceSecretBin = "${lib.getBin pkgs.replace-secret}/bin/replace-secret";
      in
      lib.mkBefore ''
        install -Dm0644 ${domainZoneFile} ${domainZoneFile'}

        ${replaceSecretBin} '#mailboxSecurityKey#' '${secretsPath "dns/${domain}/mailbox-security-key"}' '${domainZoneFile'}'
        ${replaceSecretBin} '#mailboxSecurityKeyRecord#' '${secretsPath "dns/${domain}/mailbox-security-key-record"}' '${domainZoneFile'}'
      '';
    serviceConfig.LoadCredential =
      let
        certDirectory = config.security.acme.certs."${dnsDomainName}".directory;
      in
      [
        "cert.pem:${certDirectory}/cert.pem"
        "key.pem:${certDirectory}/key.pem"
        "fullchain.pem:${certDirectory}/fullchain.pem"
      ];
  };
}

Here’s another example with modifying the Gitea service to automate certain admin tasks (i.e., setting up database schema for PostgreSQL secure schema usage, creating admin account).

Modified Gitea service with additional administration tasks
{ config, lib, pkgs, ... }:

let
  giteaDatabaseUser = "gitea";
in
{
  systemd.services.gitea = {
    # Gitea service module will have to set up certain things first which is
    # why we have to go first.
    preStart =
      let
        psqlBin = "${lib.getBin config.services.postgresql.package}/bin/psql";
        giteaBin = "${lib.getBin config.services.gitea.package}/bin/gitea";
        giteaAdminUsername = lib.escapeShellArg "foodogsquared";
      in
      lib.mkMerge [
        (lib.mkBefore ''
          # Setting up the appropriate schema for PostgreSQL secure schema usage.
          ${psqlBin} -tAc "SELECT 1 FROM information_schema.schemata WHERE schema_name='${giteaDatabaseUser}';" \
            grep -q 1 || ${psqlBin} -tAc "CREATE SCHEMA IF NOT EXISTS AUTHORIZATION ${giteaDatabaseUser};"
        '')

        (lib.mkAfter ''
          # Setting up the administrator account automated.
          ${giteaBin} admin user list --admin | grep -q ${giteaAdminUsername} \
            || ${giteaBin} admin user create \
              --username ${giteaAdminUsername} --email foodogsquared@${config.networking.domain} \
              --random-password --random-password-length 76 --admin
        '')
      ];
  };
}

It mostly requires seeing the source code for the appropriate version of nixpkgs though. But it gets easier with familiarity of navigating nixpkgs source code and nixpkgs standard library.

As for the workflow, I can push my changes easily with deploy-rs (or even just nixos-rebuild as I’ve learned it from this post). The aspect that I have trouble so far is debugging the system properly which is something I don’t know how to do…​ properly.

Before I completely decided with NixOS, I shortly ran a server with Debian which I think its a good choice for a first timer especially with its welcoming user documentations which the Nix ecosystem lacks. A lot of the documentation usually found from Nix assumes you’re familiar with the overall of a Linux system which is not a bad thing.

However, with several initiatives to make onboarding gradually better such as Zero to Nix and nix.dev, the state of Nix documentation can improve.

Once I got the ropes of managing a Debian system (for at least a week), I jumped into managing one with NixOS partly because of impatience coupled with the problems of traditional deploying and managing a system like that. While I find a lot of resources using Debian and find a solution for deploying traditional systems like Ansible or Terraform [6], I eventually find it easier to map concepts from there to its NixOS counterparts. Once the NixOS system has been setup, there’s not much of a difference except it is deployed by my computer somewhere.

The most worrying part of the setup is secrets management. It is done with sops and sops-nix then manually putting the private key in its supposed place with good ol' scp. I don’t think it’s efficient but it is what it is. I’m fairly sure there’s a better way to provision the server alongside the secrets but solving that problem seems like diminishing returns.

The maintenance process also considers budget especially in the long run so let’s be a good I.T. department for ourselves and enumerate them in a table.

Table 1. Monthly budget for running a personal cloud infrastructure
Thing Cost

A Hetzner Cloud server

€6

A Hetzner Storage Box

€4

foodogsquared.one domain (paid yearly)

€1.2

mailbox.org standard account

€3

A total of €14.2 or ₱712 in my local currency which is nice in my personal budget of under ₱1000 (€17.3).

Things to keep in mind

While creating a NixOS system has been nice, there are some things you have to keep in mind.

  • NixOS as a server works best if you’re deeply invested in the Nix ecosystem especially with the things that you can do as already discussed in The Nix ecosystem.

  • Familiarity with systemd is a must especially for extending services.

  • Extending and customizing NixOS to a similar extent as the resulting NixOS server configuration pretty much requires familiarity and some caution in keeping up-to-date with the changes to that part of nixpkgs. [7]

  • Learning to use these services in NixOS is not necessarily the same as using and configuring it as intended from upstream. You’re using it rather in the way NixOS/nixpkgs intended to.

    This could be argued the same for other operating systems such as in Debian where certain things are already handled for us through the use of debconf which makes certain tasks easier to start (e.g., setting up an OpenLDAP server, a web server).

    However, you’re still likely to run into problems such as misunderstanding and misconfigurations, both of which means referring to the documentation. Not to mention, if you really want to learn how a tool works while using NixOS, you could see the source code of the module.

Conclusion

Overall, the process of managing servers with NixOS is a great experience so far especially with how much controls it gives you from it. It sparks joy the same way how I build a NixOS system for my desktop driver. However, the investment required just for this solution is quite high which is a shame considering how simple it really is once you’ve gained some familiarity with Nix compared to other solutions such as containers. This high investment exists due to how different [8] it is compared to the rest of the ecosystem.

While it is not the de-facto experience for managing a Linux server especially for a first-timer, it can be a gateway for managing servers in a fun way sort-of-deal. There are still some things I wonder how some things work such as scaling which is Kubernetes is good at. It isn’t completely needed for me right now but it is something to ponder when considering expansion for my server fleet.

All of these are new things to me but I’m expanding what I can manage. Other plans include managing a Borg server from a BuyVM instance especially with its block storage slabs which seems to be quite flexible compared to Hetzner Storage Boxes.

Among other things, I might add a more convenient way of deploying servers from cloud providers with Terraform. Xe Iaso already has a post detailing this kind of setup but I’ll have to adjust it to deploy to Hetzner Cloud if possible. While the additional setup is more of a nuisance since it is already deployed (for the most), it is still something that can be worth in the future.


1. Very fitting note as in that previous post I also wrote my experience using NixOS as a desktop driver also in 2 months of usage. I didn’t plan for this, I swear.
2. Complete declarative configuration! MUAHAHAHAHA! Although, I don’t use it much these days but I kept it because it’s neat.
3. Which is an entire world in and of itself.
4. Which I only tried once so I can’t comment much on it. Also, the situation that needs it is very very specific so I didn’t get to use it that much.
5. sops-nix is mainly integrated with Age and GPG keys, though.
6. Though, I did found out about deploying Terraform with NixOS and there is a way to deploy one with a flakes-enabled system.
7. I’m using NixOS unstable branch so it’s more likely to bring changes and it is mostly mine to blame for using like that.
8. …​and the amount of abstractions from nixpkgs on top of Nix.