Speeding up NixOS package search on the terminal

For those who are using the new command-line interface of Nix, you’ve most likely went through to pain of searching packages through nix search. For reference, here’s what the experience looks like searching for a package from nixpkgs except it always like that every time you run the command. [1]

A part of the nix search nixpkgs experience which took 8 minutes in total to complete the whole search

It’s painful to use to the point where people tend to recommend search.nixos.org or MyNixOS for searching packages. [2] Ideally, searching packages for your system shouldn’t rely on online services. Luckily for us, there are ways how to mitigate against that.

Ezran

I’m just going to say it. That package search result looks ugly.

foodogsquared

That might be another reason why nix search is not much used.

We’re using Nix v2.18 for this post. Everything that is featured here is considered to be experimental: from flakes, the command-line interface, and even the referenced manual may change over time.

In order to enable them in the first place, you’ll have to add experimental-features = nix-command flake into the Nix configuration (see nix.conf options for more details).

But first, let’s take a closer inspection as to why nix search is behaving like that. Let’s look at the following command which is one of the examples from the manual.

nix search nixpkgs blender

At first glance, most people expect nix search nixpkgs blender searches for package blender from their nixpkgs instance. It does work as intended which is searching through packages from nixpkgs containing the word blender. Just not our own nixpkgs instance. It just doesn’t behave to how most people expect as they overlook one thing about flakes: the registry. Per the manual:

Flake registries are a convenience feature that allows you to refer to flakes using symbolic identifiers such as nixpkgs, rather than full URLs such as git://github.com/NixOS/nixpkgs. You can use these identifiers on the command line (e.g. when you do nix run nixpkgs#hello) or in flake input specifications in flake.nix files. The latter are automatically resolved to full URLs and recorded in the flake’s flake.lock file.

— Nix manual

In short, nixpkgs from nix search nixpkgs blender points to a flake from the registry. We can view what nixpkgs points to by running nix registry list.

Results from nix registry list | grep flake:nixpkgs
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable

As stated from the manual, the format from the nix registry list output comes in the form of <type> <from> <to> where…​

  • type denotes which registry it came from (which we’ll discuss shortly later in this post).

  • from typically has the flake identifier.

  • to refers to the flake reference it points to (e.g., github:NixOS/nixpkgs, github:nixos-community/home-manager).

We can see that nixpkgs points to the nixpkgs-unstable branch. That is, nix search nixpkgs blender is pretty much the same as the following command.

nix search github:NixOS/nixpkgs/nixpkgs-unstable blender

This seems fine except the flake reference isn’t pointing to a fixed point like a specific commit, it points to a branch of a remote Git repo which can change over time.

So every time you run the command, nixpkgs resolves to the most recent version of nixpkgs-unstable. This is why it downloads and evaluates a new nixpkgs instance every time it runs (or between every few hours, which is the pace for nixpkgs-unstable updates). Not exactly a good experience for a system package search compared to other operating systems like Arch Linux, Fedora, or OpenSUSE.

Ezran

Seem like flake registry is not a convenience feature at all. More like a hindrance, really.

foodogsquared

It is a CONVENIENCE FEATURE! I use it to quickly test and run packages from my nixpkgs instance as well as other projects with flake.

It’s just that most NixOS systems are not properly configured with it by default. You’ll see what I mean.

Pinning nixpkgs to the registry

Most would expect to search packages in their nixpkgs instance of their NixOS system. As hinted from the previous section, we can make use of multiple registries. As stated right after the previous excerpt from the manual:

There are multiple registries. These are, in order from lowest to highest precedence:

  • The global registry, which is a file downloaded from the URL specified by the setting flake-registry. It is cached locally and updated automatically when it’s older than tarball-ttl seconds. The default global registry is kept in a GitHub repository.

  • The system registry, which is shared by all users. The default location is /etc/nix/registry.json. On NixOS, the system registry can be specified using the NixOS option nix.registry.

  • The user registry ~/.config/nix/registry.json. This registry can be modified by commands such as nix registry pin.

  • Overrides specified on the command line using the option --override-flake.

— Nix manual

A quick way to show this would be pinning nixpkgs from the global registry to the user registry with nix registry pin nixpkgs.

Results from nix registry list | grep 'flake:nixpkgs' after pinning
user   flake:nixpkgs github:NixOS/nixpkgs/ec750fd01963ab6b20ee1f0cb488754e8036d89d
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable

Since we’re going to show configuring the system registry with NixOS, don’t forget to remove the entry from the user registry with nix registry remove nixpkgs.

Both the global and system registry can be configured in the NixOS system. In this case, we’ll be focusing on adding our flake inputs of our NixOS system into the system registry.

nix.registry expects an attribute set of flake inputs. And the outputs attribute from our flake can be a function that expects an attribute set of flake inputs returning an attribute set. We can basically set the system registry like so.

Showing how to set nix.registry
{
  description = "Very simple sample of a Nix flake with NixOS and home-manager";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-stable.url = "github:NixOS/nixpkgs/nixos-23.05";
    nixpkgs-unstable.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  };

  outputs = inputs@{ nixpkgs, ... }: {
    nixosConfigurations.desktop = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ({ config, lib, ... }: {
          nix.registry = lib.mapAttrs (_: flake: { inherit flake; }) inputs;
        })
        ./hosts/desktop
      ];
    };
  };
}

Once we rebuild our NixOS system with nixos-rebuild, we should now see our flake inputs included in the system registry.

Results from nix registry list | grep '^system'
system flake:nixpkgs path:/nix/store/kcmipm57ph9bpzz8bs80iiijiwbyzwy3-source?lastModified=1699099776&narHash=sha256-X09iKJ27mGsGambGfkKzqvw5esP1L/Rf8H3u3fCqIiU%3D&rev=85f1ba3e51676fa8cc604a3d863d729026a6b8eb
system flake:nixpkgs-stable path:/nix/store/3s69yxbbl116zwga3i6cy7prplywq0bn-source?lastModified=1699291058&narHash=sha256-5ggduoaAMPHUy4riL%2BOrlAZE14Kh7JWX4oLEs22ZqfU%3D&rev=41de143fda10e33be0f47eab2bfe08a50f234267
system flake:nixpkgs-unstable path:/nix/store/mj0hy52z22q5gpsf33akndxiclxd8ray-source?lastModified=1699343069&narHash=sha256-s7BBhyLA6MI6FuJgs4F/SgpntHBzz40/qV0xLPW6A1Q%3D&rev=ec750fd01963ab6b20ee1f0cb488754e8036d89d
system flake:self path:/nix/store/v68idbapq3m8sz0fds66vzgg7agg10g9-source?lastModified=1699449415&narHash=sha256-Whc5OQzHTJtyBbnFsDAzdjICWK2BnCDCBP6s%2Bk/oLGQ%3D&rev=e1a0efea49b2e22055d2454e76f3d01ae42fde07&revCount=3
The self flake input

Setting flake inputs to the system registry that way is convenient. But as shown from the results, there is a slight oversight with the flake input self which is a special named flake input for the current flake. Depending on your case, this can be convenient or not. At most, it is just left there…​ existing.

If you want, you can remove it like in the following snippet.

nixpkgs.lib.nixosSystem {
  modules = [
    ({ config, lib, ... }: {
      nix.registry = lib.mapAttrs
        (_: flake: { inherit flake; })
        (lib.attrsets.removeAttrs inputs [ "self" ]);
    })
  ];
}

Otherwise, you’ll have to give the self flake input from the registry a different name if you want to use it in your other flakes like in the following snippet.

Different flake.nix
{
  inputs.config.url = "self";
  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";

  # outputs...
}

If we run nix search nixpkgs blender once again, it should still evaluate but it should be done only once since:

  • We have nixpkgs pinned to our nixpkgs instance.

  • We’re fully taking advantage of flake evaluation caching meaning subsequent package searches should be (almost) instantaneous.

By configuring our registry through NixOS, we’re only evaluating nixpkgs once until we update our flake inputs with nix flake update.

Ezran

So instead of suffering every time you run the command like in the video at the beginning, you suffer at least once every update.

As another bonus, we can now take advantage of other nix subcommands such as nix build, nix shell, or nix develop.

# These commands should now use our nixpkgs instance. Hoorah!
nix run nixpkgs#neofetch
nix develop nixpkgs#gcc
nix build nixpkgs#vale
nix search nixpkgs asciidoctor

# From the previous example, we can quickly run packages from other nixpkgs
# branch easily.
nix shell nixpkgs-stable#hugo
nix shell nixpkgs-unstable#hugo
Setting user registry in home-manager

We could also set the user registry with home-manager. It even has the same option attribute as the NixOS system (i.e., nix.registry) so you could just set it once like in the following snippet.

flake.nix
{
  description = "Very simple sample of a Nix flake with NixOS and home-manager";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixpkgs-stable.url = "github:NixOS/nixpkgs/nixos-23.05";
    nixpkgs-unstable.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ nixpkgs, home-manager, ... }: let
    system = "x86_64-linux";
    nixSettings = { config, lib, pkgs, ... }: {
      # Setting each of the flake inputs as part of the system registry
      # including our own flake which is just renamed from "self" to
      # "config".
      nix.registry =
        lib.mapAttrs'
          (name: flake:
            let
              name' = if (name == "self") then "config" else name;
            in
            lib.nameValuePair name' { inherit flake; })
          inputs;
    };
  in
  {
    nixosConfigurations.desktop = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        nixSettings
        ./hosts/desktop
      ];
    };

    homeConfigurations.foodogsquared = home-manager.lib.homeManagerConfiguration {
      pkgs = nixpkgs.legacyPackages.${system};
      modules = [
        nixSettings
        ./users/foodogsquared
      ];
    };
  };
}

Take note, you cannot modify the user registry with nix registry subcommands once home-manager fully manages it.


1. Not necessarily, just every few hours as you’ll see why.
2. They also include searching for NixOS options so that may be another reason these tools are always recommended.