Musings

Packaging DS Homebrew with Nix

I recently got a DSpico, which is an open-source DS/i flashcart. While it came pre-flashed, the software behind it is under active development, so I wanted to be able to build the firmware and executables needed to bootstrap the device from source (ish). There's a guide published that goes over the steps, which involves building various tools, combining with DS firmware dump collateral, and using multiple toolchains. This is an interesting candidate for nix-ification, since it would provide infrastructure to reproducibly build firmware releases, and ensures long term stability with library/toolchain version pinning.

Toolchains

There are three targets that we need to build for:

  1. The host system, for a .NET application and some C utilities
  2. The RP2040 on the flashcart, for actually doing the flashcart things
  3. The DS itself, to run the loader/launcher applications on the flashcart.

The host toolchain is straightforward and we can simply use the nix stdenv to build most of the tools. Similarly, the RP2040 toolchain is simply gcc-arm-embedded (which is in nixpkgs) plus the pico-sdk.

The DS toolchain is much more difficult to acquire. DSpico uses the "Wonderful Toolchain" which is a homebrew focused distribution of GCC and binutils with some patches. The toolchain comes prebuilt for easy install and is designed to be entirely self-contained. It uses a fork of pacman to manage dependencies and download the builds, and is centered around a /opt/wonderful folder.

We could try to build the toolchain from source, but this seems Hard. Nix does a lot of stuff to make the compilers "nix-capable". On top of that, building a toolchain from source is an involved process with a delicate dance of binutils, gcc, and libc.

Instead, we're gonna cheat. The downstream DS SDK (BlocksDS) provides a prebuilt Docker container. If we extract the contents of that container, patch out the RPATH and linker to use nix instead, and modify the shebangs in any scripts, we should be able to make the binaries work inside of a nix build environment.

This approach is what devkitNix takes. devkitPro is much more complex to build from source and is actively hostile towards users who do so (support is only provided if you use the prebuilt binaries). devkitNix avoids the hassle by extracting all of the contents of the container and patching executables.

You can see this approach here. Broadly, the steps taken:

  1. Extract the docker layers into the build directory.
  2. Remove unneeded files to save space
  3. Patch Lua scripts to use the proper shebang (replaced /opt/wonderful with $out)
  4. Copy the opt folder to $out/opt, and create a symbolic link from $out/bin to $out/opt/bin
  5. Inform the AutoPatchelfHook about the search paths for libraries.
  6. AutoPatchelfHook will run patchelf on all the binaries it finds, replacing the paths with the new location of the libraries.

Lastly, we need to set some environment variables so that the programs we build can find the toolchain. At the time, I copied the approach devkitNix took and used stdenvAdapters.addAttrsToDerivation, but this has flaws I fixed later.

This approach yields a new stdenv-like construct with the environment variables set correctly. Thus most packages that are make based will work without further modification!

Indeed, we can build pico-launcher with ease:

{
  lib,
  stdenvDS,
  fetchFromGitHub,

}:
stdenvDS.mkDerivation (finalattrs: {
  pname = "pico-launcher";
  version = "1.1.0";
  src = fetchFromGitHub {
    owner = "LNH-team";
    repo = "pico-launcher";
    rev = "v${finalattrs.version}";
    hash = "sha256-kcn2LfgCfYh7x6Ipnw2VBZY6HvKah60r8DjEIXSiYlc=";
    fetchSubmodules = true;
  };
  enableParallelBuilding = true;

  installPhase = ''
    mkdir -p $out/root
    cp -r _pico $out
    cp LAUNCHER.nds $out/_picoboot.nds
  '';
  meta = {
    description = "Rom browser frontend for Pico Loader";
    homepage = "https://github.com/LNH-team/${finalattrs.pname}";
    license = with lib.licenses; [
      zlib
    ];
    platforms = lib.platforms.linux;
  };
})

Fantastic. Most of the programs translate easily now. One problem is that the stdenvAdapters approach prevents adding additional nativeBuildInputs. This is also pretty nonstandard to how other tools like meson work in nixpkgs. Additionally, we are not building everything we could because the SDK itself is being pulled from the docker container. Finally, we have no control over the versioning of each package going into the container.

A second approach.

Wonderful Toolchain provides a pacman repo to distribute the software. What if we could use that repo as a way to download and extract the binaries we wanted?

This approach has a few advantages. For one, we wouldn't be dumping a full FHS linux environment into the nix store with the docker container. Additionally, we could deterministically add more packages, or skip packages we don't need.

The pacman db format is fairly straightforward. Each package has a folder with a desc file inside. The desc file looks something like: 

%FILENAME%
wf-lua-5.4.8-5-x86_64.pkg.tar.xz

%NAME%
wf-lua

%BASE%
wf-lua

%VERSION%
5.4.8-5

%DESC%
Lua scripting language

%CSIZE%
243968

%ISIZE%
1385408

%SHA256SUM%
ded72e754f6882a4199cc5a412cac987c81c66d03464a7685d3ef03c83a5bdf2

...etc

We can write a simple parser to extract some of this information. We care about the name, version, arch, and sha256sum. We can use the first three to generate the URL to the release tarball, and the checksum to validate it (so nix doesn't complain).

A simple bash script can be used to parse the pacman database and produce a JSON file which is then parsed by nix.

We can describe a package that takes a list of pacman packages as an input and produces a derivation where the packages have been all extracted into a single location and patched:

# given a package name, extract and construct a derivation
# based on the source information in the package name
{
  # list of packages to stack
  wonderfulPackages ? [ ],
  autoPatchelfHook,
  writeText,
  lib,
  fetchurl,
  pkgsMusl,
}:
let
  allPackageAttrs = builtins.fromJSON (builtins.readFile ./packages.json);

  # filter packages based on name being in wonderfulPackages
  pkgs = lib.attrsets.getAttrs wonderfulPackages allPackageAttrs;

  mkPkgSrc = (
    name: spec:
    fetchurl {
      url = "https://wonderful.asie.pl/packages/rolling/linux/x86_64/${name}-${spec.version}-${spec.arch}.pkg.tar.xz";
      hash = spec.sha256sum;
    }
  );
  srcs = (lib.attrsets.mapAttrsToList (mkPkgSrc) pkgs);
in
pkgsMusl.stdenv.mkDerivation (finalAttrs: {
  name = "wonderful-toolchain";
  inherit srcs;

  nativeBuildInputs = [
    autoPatchelfHook
  ];
  buildInputs = [
    pkgsMusl.stdenv.cc.cc.lib
    pkgsMusl.zstd
    pkgsMusl.zlib
  ];
  dontBuild = true;
  dontConfigure = true;
  patchPhase = ''
    runHook prePatch
    rm .BUILDINFO .MTREE .PKGINFO
    runHook postPatch
  '';
  installPhase = ''
    runHook preInstall
    ls -lah
    mkdir -p $out/wonderful
    cp -r . $out/wonderful/
    if [[ -d $out/wonderful/bin ]]; then
      ln -sf $out/wonderful/bin $out/bin
    fi
    addAutoPatchelfSearchPath $out/wonderful/lib
    runHook postInstall
  '';

  setupHook = writeText "wonderful-hook.sh" ''
    export WONDERFUL_TOOLCHAIN=@out@/wonderful
  '';

  sourceRoot = ".";

})

Some tweaks were required to make this work:

  1. sourceRoot = ".". Pacman packages are simply unpacked to / to install normally. sourceRoot sets the build directory when multiple sources are provided. Setting it to . means that every package will be extracted and we will be at the root of all of those extracted packages.
  2. pkgsMusl. Wonderful toolchain uses musl as its host libc, but dynamically linked which is a little unusual. The autoPatchelf hook will handle linking these to the wonderful binaries.
  3. seupHook. This is rather poorly defined in nixpkgs but allows packages to modify the build environment when loaded as a nativeBuildInput. In this case, it simply sets the WONDERFUL_TOOLCHAIN variable to point to our build.

This was very cool. Even though it's not great to be downloading binaries, the language features of nix make it easy to do so in a pinch. We also unlock the ability to compile BlocksDS from source and integrate it into nix. We use a similar setupHook trick to set the environment of packages that depend on it.

build the rest of the flashcart

We've got the toolchain, now we just need the rest of the flashcart.

The dependency graph is a little confusing here. I basically went down the provided guide and took any package and made an appropriate nix derivation. There are some weird ones, like the pico-loader project which has both a .NET package and a DS program built with the same Makefile.

wonderful-toolchain nintendo-collateral blocksds DSRomEncryptor dspico-dldi bootloader wrfuxxed pico-loader firmware pico-launcher rootfs

the build graph we need

When making a package set that is interdependent, you have to do shenanigans to callPackage to make it aware of the new packages you are adding. This can either be done with callPackageWith or by making an overlay for nixpkgs.

The callPackageWith method, per nix.dev:

let
  pkgs = import <nixpkgs> { };
  callPackage = pkgs.lib.callPackageWith (pkgs // packages);
  packages = {
    a = callPackage ./a.nix { };
    b = callPackage ./b.nix { };
    c = callPackage ./c.nix { };
    d = callPackage ./d.nix { };
    e = callPackage ./e.nix { };
  };
in
packages

My approach, using an overlay:

final: prev:
prev.lib.packagesFromDirectoryRecursive {
  callPackage = final.callPackage;
  directory = ./packages;
}
// {
  wonderful-toolchain = final.callPackage ./wonderful-toolchain {
    wonderfulPackages = [
      "toolchain-gcc-arm-none-eabi-binutils"
      "toolchain-gcc-arm-none-eabi-gcc"
      "toolchain-gcc-arm-none-eabi-gcc-libs"
      "toolchain-gcc-arm-none-eabi-picolibc-generic"
      "toolchain-gcc-arm-none-eabi-libstdcxx-picolibc"
    ];
  };
};

By using final.callPackage, our packages have already been added to the package set and will resolve correctly. The lazy evaluated nature of Nix means that references that seem self-referential are fine so long as they are not cyclical. This nuance is difficult to convey.

nintendo collateral

Some of the steps require the user to provide files which are taboo to distribute. Since we need those files, we should provide a spot for the user to place them. We can verify their hashes so that the build is still stable. I have a nintendo-collateral derivation which simply verifies the hash and copies the files into the nix store.

I also just put the files into git anyways. The console is 20 years old.

Conclusion

This project demonstrates some of the more confusing details of nix packaging. I learned more about callPackage, setupHook (which I still don't fully understand, especially in relation to envHook), and patchelf, which is used to package binary packages in nixpkgs. I feel much more comfortable packaging software in nix for the most part. The final repo lives here.

Someday I'll revisit this and build the toolchain from source myself, though that will probably be a post of its own.