├── LICENSE ├── README.md ├── flake.lock ├── flake.nix ├── garnix.yaml └── keys ├── ssh_host_ed25519_key └── ssh_host_ed25519_key.pub /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2022 Gabriella Gonzalez 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | 1. Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | 2. Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | 3. Neither the name of the author nor the names of its contributors may be 15 | used to endorse or promote products derived from this software without 16 | specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bootstrap a Linux build VM on macOS 2 | 3 | **NOTE:** This has now been 4 | [upstreamed into Nixpkgs](https://nixos.org/manual/nixpkgs/unstable/#sec-darwin-builder) 5 | but I keep this repository around for backwards compatibility and historical reference. 6 | 7 | This repository provides a way to bootstrap a NixOS Linux build VM running on 8 | macOS without relying on an existing NixOS builder. You can then in turn use 9 | that NixOS build VM to to build and run other NixOS VMs on macOS. 10 | 11 | ## Requirements 12 | 13 | This requires macOS version 12.4 or later and Nix version 2.4 or later. 14 | 15 | This also requires that port 22 on your machine is free (since Nix does not 16 | permit specifying a non-default SSH port for builders). 17 | 18 | You will also need to be a trusted user for your Nix installation. In other 19 | words, your `/etc/nix/nix.conf` should have something like: 20 | 21 | ``` 22 | extra-trusted-users = 23 | ``` 24 | 25 | ## Instructions 26 | 27 | Before performing any of these commands, read the following security disclaimer: 28 | 29 | * [Security - Cache](#security---cache) 30 | 31 | If you haven't already, add this to `/etc/nix/nix.conf`: 32 | 33 | ``` 34 | extra-experimental-features = nix-command flakes 35 | ``` 36 | 37 | … and then restart your Nix daemon to apply the change: 38 | 39 | ```ShellSession 40 | $ sudo launchctl kickstart -k system/org.nixos.nix-daemon 41 | ``` 42 | 43 | Then launch a macOS builder by running this command: 44 | 45 | ```ShellSession 46 | $ nix run github:Gabriella439/macos-builder 47 | ``` 48 | 49 | …and if you trust me then confirm when prompted to do so: 50 | 51 | ```ShellSession 52 | do you want to allow configuration setting 'extra-substituters' to be set to 'https://macos-builder.cachix.org' (y/N)? y 53 | do you want to permanently mark this value as trusted (y/N)? y 54 | do you want to allow configuration setting 'extra-trusted-public-keys' to be set to 'macos-builder.cachix.org-1:HPWcq59/iyqQz6HEtlO/kjD/a7ril0+/XJc+SZ2LgpI=' (y/N)? y 55 | do you want to permanently mark this value as trusted (y/N)? y 56 | ``` 57 | 58 | That will prompt you to enter your `sudo` password: 59 | 60 | ``` 61 | + sudo --reset-timestamp /nix/store/…-install-credentials.sh ./keys 62 | Password: 63 | ``` 64 | 65 | … so that it can install a private key used to `ssh` into the build server. 66 | After that the script will launch the virtual machine: 67 | 68 | ``` 69 | <<< Welcome to NixOS 22.11.20220901.1bd8d11 (aarch64) - ttyAMA0 >>> 70 | 71 | Run 'nixos-help' for the NixOS manual. 72 | 73 | nixos login: 74 | ``` 75 | 76 | … and your remote builder is good to go! When you need to stop the VM, type 77 | Ctrl-a + c to open the `qemu` prompt and then 78 | type `quit` followed by Enter. 79 | 80 | To use the builder, add the following options to your `nix.conf` file: 81 | 82 | ``` 83 | # - Replace ${ARCH} with either aarch64 or x86_64 to match your host machine 84 | # - Replace ${MAX_JOBS} with the maximum number of builds (pick 4 if you're not sure) 85 | builders = ssh-ng://builder@localhost ${ARCH}-linux /etc/nix/builder_ed25519 ${MAX_JOBS} - - - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo=' 86 | 87 | # Not strictly necessary, but this will reduce your disk utilization 88 | builders-use-substitutes = true 89 | ``` 90 | 91 | … and then restart your Nix daemon to apply the change: 92 | 93 | ```ShellSession 94 | $ sudo launchctl kickstart -k system/org.nixos.nix-daemon 95 | ``` 96 | 97 | … and you're done! Enjoy 😊 98 | 99 | ## Building downstream VMs 100 | 101 | You don't have to stop there! You can use the Linux builder you just created 102 | to build and run other NixOS VMs on macOS. Here is an example of a flake that 103 | you can use as a starting template: 104 | 105 | ```nix 106 | { inputs.macos-builder.url = "github:Gabriella439/macos-builder"; 107 | 108 | outputs = { macos-builder, nixpkgs, ... }: { 109 | nixosConfigurations.default = nixpkgs.lib.nixosSystem { 110 | modules = [ macos-builder.nixosModules.aarch64-darwin.default ]; 111 | 112 | system = "aarch64-linux"; 113 | }; 114 | }; 115 | } 116 | ``` 117 | 118 | … and you can run that NixOS VM on macOS using: 119 | 120 | ```ShellSession 121 | $ nix run .#nixosConfigurations.default.config.system.build.vm 122 | ``` 123 | 124 | ## Architecture match 125 | 126 | This only supports running a guest system of the same architecture. 127 | 128 | In other words: 129 | 130 | - an `aarch64-darwin` host can run an `aarch64-linux` guest 131 | - an `x86_64-darwin` host can run an `x86_64-linux` guest 132 | 133 | … but: 134 | 135 | - an `aarch64-darwin` host cannot run an `x86_64-linux` guest 136 | - an `x86_64-darwin` host cannot run an `aarch64-linux` guest 137 | 138 | ## Security - Cache 139 | 140 | By trusting my cache you are essentially running arbitrary code you downloaded 141 | from me. If you don't trust me, then don't confirm when prompted to trust my 142 | cache. 143 | 144 | In the long run, this will (hopefully) be built and served by `cache.nixos.org` 145 | so that you don't have to trust me. If you're patient, you can wait until that 146 | happens. 147 | 148 | If you're impatient, then you have to trust me or ask someone who you do trust 149 | that has a Linux builder to build and cache this repository for you. 150 | 151 | For what it's worth, I did *not* use the shared aarch64.nixos.community 152 | machine for building this. I provisioned a blank 153 | `NixOS-22.05.342.a634c8f6c1f-aarch64-linux` AMI to build and populate the 154 | cache. 155 | 156 | ## Acknowledgments 157 | 158 | The work in this repository is based in part on prior work from: 159 | 160 | - [NixOS/nixpkgs#108984](https://github.com/NixOS/nixpkgs/issues/108984). 161 | - [YorikSar/nixos-vm-on-macos](https://github.com/YorikSar/nixos-vm-on-macos) 162 | -------------------------------------------------------------------------------- /flake.lock: -------------------------------------------------------------------------------- 1 | { 2 | "nodes": { 3 | "flake-utils": { 4 | "locked": { 5 | "lastModified": 1659877975, 6 | "narHash": "sha256-zllb8aq3YO3h8B/U0/J1WBgAL8EX5yWf5pMj3G0NAmc=", 7 | "owner": "numtide", 8 | "repo": "flake-utils", 9 | "rev": "c0e246b9b83f637f4681389ecabcb2681b4f3af0", 10 | "type": "github" 11 | }, 12 | "original": { 13 | "owner": "numtide", 14 | "repo": "flake-utils", 15 | "type": "github" 16 | } 17 | }, 18 | "nixpkgs": { 19 | "locked": { 20 | "lastModified": 1663460020, 21 | "narHash": "sha256-rDClYj1dCXd3v5ZbIkxQLn5rkdFxqJFrPbnwz3GNXUE=", 22 | "owner": "nixos", 23 | "repo": "nixpkgs", 24 | "rev": "f1a49e20e1b4a7eeb43d73d60bae5be84a1e7610", 25 | "type": "github" 26 | }, 27 | "original": { 28 | "owner": "nixos", 29 | "ref": "master", 30 | "repo": "nixpkgs", 31 | "type": "github" 32 | } 33 | }, 34 | "root": { 35 | "inputs": { 36 | "flake-utils": "flake-utils", 37 | "nixpkgs": "nixpkgs" 38 | } 39 | } 40 | }, 41 | "root": "root", 42 | "version": 7 43 | } 44 | -------------------------------------------------------------------------------- /flake.nix: -------------------------------------------------------------------------------- 1 | { inputs = { 2 | nixpkgs.url = "github:nixos/nixpkgs/master"; 3 | 4 | flake-utils.url = "github:numtide/flake-utils"; 5 | }; 6 | 7 | outputs = { self, nixpkgs, flake-utils }: 8 | flake-utils.lib.eachSystem [ "aarch64-darwin" "x86_64-darwin" ] (system: 9 | let 10 | keysDirectory = "/var/keys"; 11 | 12 | user = "builder"; 13 | 14 | keyType = "ed25519"; 15 | 16 | in 17 | rec { 18 | defaultPackage = packages.default; 19 | 20 | packages = { 21 | default = 22 | nixosConfigurations.default.config.system.build.vm; 23 | 24 | builder = 25 | nixosConfigurations.builder.config.system.build.vm; 26 | 27 | app = 28 | let 29 | pkgs = nixpkgs.legacyPackages."${system}"; 30 | 31 | privateKey = "/etc/nix/${user}_${keyType}"; 32 | 33 | publicKey = "${privateKey}.pub"; 34 | 35 | # This installCredentials script is written so that it's as easy 36 | # as possible for a user to audit before confirming the `sudo` 37 | installCredentials = pkgs.writeShellScript "install-credentials.sh" '' 38 | KEYS="''${1}" 39 | INSTALL=${pkgs.coreutils}/bin/install 40 | "''${INSTALL}" -g nixbld -m 600 "''${KEYS}/${user}_${keyType}" ${privateKey} 41 | "''${INSTALL}" -g nixbld -m 644 "''${KEYS}/${user}_${keyType}.pub" ${publicKey} 42 | ''; 43 | 44 | in 45 | pkgs.writeShellScript "create-builder.sh" '' 46 | KEYS="''${KEYS:-./keys}" 47 | ${pkgs.coreutils}/bin/mkdir --parent "''${KEYS}" 48 | PRIVATE_KEY="''${KEYS}/${user}_${keyType}" 49 | PUBLIC_KEY="''${PRIVATE_KEY}.pub" 50 | if [ ! -e "''${PRIVATE_KEY}" ] || [ ! -e "''${PUBLIC_KEY}" ; then 51 | ${pkgs.coreutils}/bin/rm --force -- "''${PRIVATE_KEY}" "''${PUBLIC_KEY}" 52 | ${pkgs.openssh}/bin/ssh-keygen -q -f "''${PRIVATE_KEY}" -t ${keyType} -N "" -C 'builder@localhost' 53 | fi 54 | if ! ${pkgs.diffutils}/bin/cmp "''${PUBLIC_KEY}" ${publicKey}; then 55 | (set -x; sudo --reset-timestamp ${installCredentials} "''${KEYS}") 56 | fi 57 | KEYS="$(nix-store --add "$KEYS")" ${packages.builder}/bin/run-nixos-vm 58 | ''; 59 | }; 60 | 61 | defaultApp = apps.default; 62 | 63 | apps.default = { 64 | type = "app"; 65 | 66 | program = "${packages.app}"; 67 | }; 68 | 69 | nixosConfigurations = 70 | let 71 | toGuest = builtins.replaceStrings [ "darwin" ] [ "linux" ]; 72 | 73 | in 74 | { default = nixpkgs.lib.nixosSystem { 75 | system = toGuest system; 76 | 77 | modules = [ nixosModules.default ]; 78 | }; 79 | 80 | builder = nixpkgs.lib.nixosSystem { 81 | system = toGuest system; 82 | 83 | modules = [ nixosModules.builder ]; 84 | }; 85 | }; 86 | 87 | nixosModule = nixosModules.default; 88 | 89 | nixosModules = rec { 90 | vm = { modulesPath, ... }: { 91 | imports = [ "${modulesPath}/virtualisation/qemu-vm.nix" ]; 92 | 93 | # DNS fails for QEMU user networking (SLiRP) on macOS. See: 94 | # 95 | # https://github.com/utmapp/UTM/issues/2353 96 | # 97 | # This works around that by using a public DNS server other than the 98 | # DNS server that QEMU provides (normally 10.0.2.3) 99 | networking.nameservers = [ "8.8.8.8" ]; 100 | }; 101 | 102 | build = { 103 | environment.etc = { 104 | "ssh/ssh_host_ed25519_key" = { 105 | mode = "0600"; 106 | 107 | source = "${keysDirectory}/ssh_host_ed25519_key"; 108 | }; 109 | 110 | "ssh/ssh_host_ed25519_key.pub" = { 111 | mode = "0644"; 112 | 113 | source = "${keysDirectory}/ssh_host_ed25519_key.pub"; 114 | }; 115 | }; 116 | 117 | nix.settings = { 118 | auto-optimise-store = true; 119 | 120 | min-free = 1024 * 1024 * 1024; 121 | 122 | max-free = 3 * 1024 * 1024 * 1024; 123 | 124 | trusted-users = [ "root" user ]; 125 | }; 126 | 127 | services.openssh = { 128 | enable = true; 129 | 130 | authorizedKeysFiles = [ "${keysDirectory}/%u_${keyType}.pub" ]; 131 | }; 132 | 133 | system.stateVersion = "22.05"; 134 | 135 | users.users."${user}"= { 136 | isNormalUser = true; 137 | }; 138 | 139 | virtualisation = { 140 | diskSize = 20 * 1024; 141 | 142 | forwardPorts = [ 143 | { from = "host"; guest.port = 22; host.port = 22; } 144 | ]; 145 | 146 | # Disable graphics for the builder since users will likely want to 147 | # run it non-interactively in the background. 148 | graphics = false; 149 | 150 | sharedDirectories.keys = { 151 | source = "\"$KEYS\""; 152 | target = keysDirectory; 153 | }; 154 | 155 | # If we don't enable this option then the host will fail to delegate 156 | # builds to the guest, because: 157 | # 158 | # - The host will lock the path to build 159 | # - The host will delegate the build to the guest 160 | # - The guest will attempt to lock the same path and fail because 161 | # the lockfile on the host is visible on the guest 162 | # 163 | # Snapshotting the host's /nix/store as an image isolates the guest 164 | # VM's /nix/store from the host's /nix/store, preventing this 165 | # problem. 166 | useNixStoreImage = true; 167 | 168 | # Obviously the /nix/store needs to be writable on the guest in 169 | # order for it to perform builds. 170 | writableStore = true; 171 | 172 | # This ensures that anything built on the guest isn't lost when the 173 | # guest is restarted. 174 | writableStoreUseTmpfs = false; 175 | }; 176 | }; 177 | 178 | default = { 179 | imports = [ vm ]; 180 | 181 | virtualisation.host.pkgs = nixpkgs.legacyPackages."${system}"; 182 | }; 183 | 184 | builder.imports = [ default build ]; 185 | }; 186 | }); 187 | 188 | nixConfig = { 189 | extra-substituters = [ "https://macos-builder.cachix.org" ]; 190 | 191 | extra-trusted-public-keys = [ 192 | "macos-builder.cachix.org-1:HPWcq59/iyqQz6HEtlO/kjD/a7ril0+/XJc+SZ2LgpI=" 193 | ]; 194 | }; 195 | } 196 | -------------------------------------------------------------------------------- /garnix.yaml: -------------------------------------------------------------------------------- 1 | builds: 2 | exclude: [] 3 | include: 4 | - packages.aarch64-darwin.* 5 | - apps.aarch64-darwin.* 6 | - packages.x86_64-darwin.* 7 | - apps.x86_64-darwin.* 8 | -------------------------------------------------------------------------------- /keys/ssh_host_ed25519_key: -------------------------------------------------------------------------------- 1 | -----BEGIN OPENSSH PRIVATE KEY----- 2 | b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW 3 | QyNTUxOQAAACCQVnMW/wZWqrdWrjrRPhfEFFq1KLYguagSflLhFnVQmwAAAJASuMMnErjD 4 | JwAAAAtzc2gtZWQyNTUxOQAAACCQVnMW/wZWqrdWrjrRPhfEFFq1KLYguagSflLhFnVQmw 5 | AAAEDIN2VWFyggtoSPXcAFy8dtG1uAig8sCuyE21eMDt2GgJBWcxb/Blaqt1auOtE+F8QU 6 | WrUotiC5qBJ+UuEWdVCbAAAACnJvb3RAbml4b3MBAgM= 7 | -----END OPENSSH PRIVATE KEY----- 8 | -------------------------------------------------------------------------------- /keys/ssh_host_ed25519_key.pub: -------------------------------------------------------------------------------- 1 | ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBWcxb/Blaqt1auOtE+F8QUWrUotiC5qBJ+UuEWdVCb root@nixos 2 | --------------------------------------------------------------------------------