Revisiting GitHub Actions on nixfiles.
Written on Dec 27, 2024.
Merry Christmas! I have nothing better to do in holidays anyway, so why not spend the time to experiment crazy things on my production notebook? Namely, upgrading MacOS to Sequoia, and then updating my nixfiles and nixfiles-home flakes.
So basically a full system plus userspace upgrade.
The upgrade took some time and all went well, nothing broke too much except one VSCode extension gone, and then of course kitty broke again. Hydra couldn't build it for all darwin platforms [1] [2], so problem is not on my end.
I took a look and figured that it's upstream pushing broken code [3], and not the first time. In hindsight, the maintainers could have tested on other platforms, and then put in a hotfix, but then I wouldn't really expect that realistically.
It's more or less fine, as I overrideAttrs the src to kitty/master in my own config. But what's not fine is that, hey, my GitHub Actions workflow is failing and I forgot about it totally! It was broken from about two months ago.
I opened up the workflow file, stared at the ugly, gibberish code, tried to understand why I wrote it like that. Then the memories came flooding back, and reminded me why I gave up in the first place.
It's one colossal nightmare with bootstrapping and nested virtualization, and it's not pretty.
I like my nixfiles like that
My config is a bit complicated. My nixfiles uses nix-darwin, and within it has a really neat option called nix.linux-builder which allows you to start qemu-system-aarch64 with HVF acceleration, then tell the nix daemon to use that VM to build linux stuff.
It works well by itself, but it also offers a nix.linux-builder.config option for you to customize the VM config. So for me, I use it to put in my own public SSH key, so I don't have to read /etc/nix/builder_ed25519 with root every time I want to SSH into the VM. Along with many other things.
Here's the problem...
Bootstrapping
However, the new configuration in nix.linux-builder.config requires a pre-existing linux-builder to build from, otherwise it just flops and says
a 'aarch64-linux' with features {} is required to build '/nix/store/...',
but I am a 'aarch64-darwin'Even now, I haven't really found a good way to solve this.
How I worked around it in the past was to manually comment out the nix.linux-builder.config and build the default, cached linux-builder first, then finally the system can use that to build the nix.linux-builder.config.
But of course that's not really a solution to GitHub Actions, because I can't tell it comment out my code like that automagically.
Nested virtualization
Here's a bigger problem: GitHub Actions doesn't support nested virtualization, and in fact such things are even worse on Apple Silicon, because only recently did they add support for M3 and later. So we can't use accel=hvf:tcg like in the sane defaults, or qemu will explode with invalid accelerator hvf.
So, not only we need to instruct the VM to bootstrap the linux-builder first, we also need to override the nix.linux-builder.package to hell and back so it won't try to use HVF acceleration.
The eventual solution
Coupled with my efforts two months ago, I stayed up a night and came up with an even uglier (but works™️) solution.
We basically tell the runner to bootstrap itself with a default nix.linux-builder, but overridden to run without HVF. It's not pretty, but it works.
{ lib, pkgs, ... }:
{
nix.linux-builder.enable = true;
nix.linux-builder.package = lib.makeOverridable ({ modules }: (
pkgs.runCommand "linux-builder-with-no-hvf-for-github-actions" {
nativeBuildInputs = [ pkgs.darwin.linux-builder ];
} ''
create_builder_orig="${pkgs.darwin.linux-builder.override { inherit modules; }}/bin/create-builder"
run_nixos_vm_orig=$(grep -o '/nix/store/[a-z0-9-]*/bin/run-nixos-vm' "$create_builder_orig")
mkdir -p $out/bin
sed "s|,accel=hvf:tcg||g;s|smp 1|smp $(nproc)|g" "$run_nixos_vm_orig" > "$out/bin/run-nixos-vm"
sed "s|$run_nixos_vm_orig|$out/bin/run-nixos-vm|g" "$create_builder_orig" > "$out/bin/create-builder"
chmod +x "$out/bin/run-nixos-vm" "$out/bin/create-builder"
''
) // pkgs.darwin.linux-builder.passthru) { modules = [ ]; };
}We override the nix.linux-builder.package to essentially copy over two scripts from the original linux-builder package, then remove the HVF acceleration from run-nixos-vm and patch create-builder to use our new run-nixos-vm instead, while all other things are left untouched.
I didn't have a spare machine to run GitHub Actions on, so I just force-pushed and hoped for the best.
It succeeded with workflow run #207. Ah, just growing pains with nix.