Deploying NixOS with Rollback

Date: 2026-06-01
tags: nixos

One very cool feature of NixOS is nixos-rebuild --target-host. Essentially, it builds a new NixOS configuration and pushes it to a remote SSH target and switches that target to this new configuration. Think of it like an Ansible playbook for your servers, but "for free". Or terraform, if terraform was good.

As with a local configuration change, you can either switch to the new configuration now, or use it on the next boot. Typically you can just use switch, but when doing larger upgrades it's sometimes necessary to use boot instead.

One issue of this tool is that if you screw up, your remote system becomes inaccessible. If you push a bad network or SSH server configuration, you'll have to either walk over to the machine or use some out-of-band management to roll back to a working build. This can be very frustrating if the server is remote, or doesn't have a BMC of some kind.

This problem is similar to one faced on fancy networking equipment. It's often the case that a change needs to be deployed remotely in a large network. But changing network configurations while using the same network to do so can result in a lockout like above. To solve this, most router/switch OSes let you test a configuration before confirming the change. If you don't confirm in time (i.e you got disconnected), it will automatically roll back to the previous configuration.

This has several nice properties:

Rollback is initiated on the device itself, not done remotely.
It doesn't try to assume what "working" means: even if you can still access the device, other things might be broken. Or things might show errors but you don't care or want to inspect them.
It can be automated in cases where it should probably just work.

For deploying NixOS, one popular tool is deploy-rs, which has a "Magic Rollback" feature like the above. It has a few differences:

rollback and confirmation is automatic if we can connect and no systemd services fail
it only works when switching, not when trying to boot into a new configuration.

And crucially, I don't really understand how it works. There's some rust binary that gets built for the remote system, which complicates the cross-compilation stuff.

I think we can make a better system that is easier to understand with some scripting.

How NixOS Configurations Work

Before we talk about reverting a NixOS configuration change, we need to learn how NixOS configurations work. The typical program is nixos-rebuild. This was historically a giant bash script but is now Python. You can read the relevant source here. Essentially it boils down to three operations:

Realize the new configuration in the nix store, either by building it or copying it.
Update the system profile to point to this new NixOS configuration in the store.
Call <configuration>/bin/switch-to-configuration $mode where $mode is switch or boot. This is what is referred to as "activation".

It's important to note that <configuration>/bin/ is not the same as the /bin folder in the resulting NixOS environment, but instead just has this activation script inside. The script is built to then do various things that make the current system correct.

Fundamentally all other nixos-rebuild options are just slight modifications to this flow. The exception is build-image, which realizes a different attribute for the configuration and calls a script that builds an image using qemu instead.

With this covered, we can start to work on persistent automatic rollbacks using systemd.

Rolling back

We want the following flow:

Activate a new generation of NixOS, and set a flag.
If X time passes and the flag is still present, rollback.
If the flag is cleared, don't do anything (success)

We want this flag to be persistent. One way I thought of is to use systemd's path units to then activate a timer unit, which then finally activates a service unit. The service unit would actually execute the code that would roll the system back.

file exists --> path watcher activates --> timer activates --> service

To stop the rollback, we would delete the file and also stop the timer/service units. They would reactivate once the file is placed again, since the path unit is still active.

Since we have a file to store stuff in, we can also put the desired "mode" of the operation as the contents (either switch or boot). This way we know if we need to reboot to roll back cleanly.

We'll have three scripts on the system:

nixos-rollback-commit switches to the configuration and sets up the auto rollback.
nixos-rollback-confirm removes the rollback flag, keeping the current configuration.
nixos-rollback executes the rollback.

If the activation mode is "boot", we'll temporarily the rollback for the current boot, but it will start the timer on the next boot. The way to do this is using systemd mask --runtime <unit>. Masking a systemd unit prevents it from running. The --runtime flag causes this to only block during the current systemd session.

This should be everything. Let's define a rollback flag at /var/lib/rollback/mode, and write the nixos-rollback-commit script.

rollback-mode = "${cfg.basePath}/mode";
nixos-rollback-commit = pkgs.writeShellApplication {
  name = "nixos-rollback-commit";
  runtimeInputs = with pkgs; [
    config.nix.package
    systemd
  ];
  text = ''
    if [ $# -ne 2 ]; then
      echo "usage: nixos-rollback-commit <path> mode"
      echo "mode=boot/switch"
      exit 1
    fi
    mode=$2

    if [ "$mode" != "boot" ] && [ "$mode" != "switch" ]; then
      echo "mode must be boot/switch"
      exit 1
    fi
    if [ ! -d "$1" ]; then
      echo "outpath does not exist or is not a directory"
      exit 1
    fi
    if [ -f "${rollback-mode}" ]; then
      echo "A rollback has already been started."
      echo "Run nixos-rollback-confirm to delete it"
      exit 1
    fi


    if [ "$mode" = "boot" ]; then
      echo "boot mode: Auto-Rollback timer will start on next boot."
      systemctl mask --runtime ${name}.path
    else
      echo "Auto-Rollback enabled. System will rollback in ${builtins.toString cfg.rollbackDelay} seconds."
    fi
    # These lines are the only ones that matter.
    echo -n "$mode" > "${rollback-mode}"
    echo "Activating new configuration..."
    nix-env -p /nix/var/nix/profiles/system --set "$1"
    "$1/bin/switch-to-configuration" "$mode"
  '';
};

This script takes an output path (outpath) and a mode as the arguments. Fundamentally it is just saving the mode to our dedicated location (which will activate the path unit) and doing the profile and activation of the new configuration. There's some logic to prevent double-activation, and the boot mode will disable the path unit until the next boot.

Next, let's write the confirmation script:

nixos-rollback-confirm = pkgs.writeShellApplication {
  name = "nixos-rollback-confirm";
  runtimeInputs = with pkgs; [
    config.nix.package
    systemd
  ];
  text = ''
    if [ ! -f "${rollback-mode}" ]; then
      echo "No rollback in place"
      exit 1
    fi
    rm ${rollback-mode}
    systemctl stop ${name}.timer || true
    systemctl stop ${name}.service || true
    systemctl unmask --runtime ${name}.path || true
  '';
};

This one is fairly simple. Remove the rollback file if it exists, and then stop the timer and the service. We also unmask the path unit.

Finally, the rollback script that gets executed if the timer triggers:

nixos-rollback = pkgs.writeShellApplication {
  name = "nixos-rollback";
  runtimeInputs = with pkgs; [
    config.nix.package
    systemd
  ];
  text = ''
    if [ ! -e "${rollback-mode}" ]; then
      echo "Rollback is not active."
      exit 0
    fi
    mode=$(cat "${rollback-mode}")
    echo "rolling back with $mode"
    nix-env --profile /nix/var/nix/profiles/system --rollback
    "/nix/var/nix/profiles/system/bin/switch-to-configuration" "$mode"

    rm -f ${rollback-mode}

    if [ "$mode" = "boot" ]; then
      shutdown -r now "Rebooting after rollback"
    fi
  '';

};

Notably we reboot if the mode we used was "boot", since that's the only way to actually go back to the previous revision safely.

We combine these scripts into a NixOS module that installs them and also sets up the systemd units:

# ... scripts defined up here in a `let` block
{
  options.my.rollback = with lib; {
    enable = mkEnableOption "Enable rollback capabilities";
    basePath = mkOption {
      description = "Where to store the rollback state";
      type = with types; path;
      default = "/var/lib/rollback";
    };
    rollbackDelay = mkOption {
      description = "Delay before rolling back (seconds)";
      type = with types; int;
      default = 180;
    };
  };

  config = lib.mkIf cfg.enable {
    environment.systemPackages = [
      nixos-rollback-confirm
      nixos-rollback-commit
      nixos-rollback
    ];

    systemd.paths."${name}" = {
      description = "Path watcher for rollback";
      # very early startup
      wantedBy = [ "sysinit.target" ];
      pathConfig = {
        PathExists = rollback-mode;
        # this is elaborate but it's just saying "${name}.timer"
        Unit = config.systemd.timers."${name}".name;
      };
    };

    systemd.timers."${name}" = {
      description = "NixOS Rollback timer";
      timerConfig = {
        Unit = config.systemd.services."${name}".name;
        OnActiveSec = cfg.rollbackDelay;
      };
    };
    systemd.services."${name}" = {
      description = "NixOS rollback detector";
      script = "${nixos-rollback}/bin/nixos-rollback";
      serviceConfig = {
        User = "root";
      };
    };
    systemd.tmpfiles.settings."10-rollback" = {
      "${cfg.basePath}".d = {
        user = "root";
        group = "root";
        mode = "0755";
      };
    };
  };

}

One quirk to note is that systemd will automatically associate the service with the timer since they have the same name. The path unit would normally also activate the service by default, so we have to manually tell it to activate the timer instead (the Unit option in the pathConfig).

With this, we've got a basic rollback scheme that works across reboots that can exist in a single file. Using these tools is a little strange since it replaces part of nixos-rebuild. But essentially:

nix-build -A nixosConfigurations.remoraid.config.system.build.toplevel
nix copy --to ssh://root@remoraid result/ 
ssh root@remoraid nixos-rollback-commit $(realpath result) switch # or boot
ssh root@remoraid nixos-rollback-confirm

In a later post I might elaborate and publish my autodeploy python script which combines this with a few other features.

Conclusions

Sometimes, it's better to take a simpler approach. There are cases where deploy-rs is the correct choice, but for me it was opaque and I didn't understand how it worked. Now that I've made my own, I can see what profiles and "activation" mean in the NixOS context, but also can customize the flow to fit my needs. Since we didn't use any compiled programming language, there's no additional cross compilation burden.

This system isn't as keen to put systemd errors in your face, which is sometimes an issue. Perhaps being able to define "health checks" using Python or bash would be a good solution. This is left as an exercise to the reader.