Skip to content

Instantly share code, notes, and snippets.

@samelie

samelie/utm-nix.md

Created November 13, 2025 17:43
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save samelie/db65e7decbfdb74d748d44860840b51f to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save samelie/db65e7decbfdb74d748d44860840b51f to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. samelie created this gist Nov 13, 2025.
    630 changes: 630 additions & 0 deletions utm-nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,630 @@
    # NixOS on UTM for Apple Silicon: Portable Configuration Research

    ## Overview
    Creating portable, automated NixOS VMs on Apple Silicon Macs using UTM virtualization.

    ## Key Projects

    ### 1. ciderale/nixos-utm ⭐21
    **URL**: https://github.com/ciderale/nixos-utm
    **Purpose**: Automate creation of UTM-based NixOS VMs
    **Key Features**:
    - Wraps `nixos-anywhere` to eliminate manual provisioning steps
    - Apple Virtualization backend support w/ Rosetta2 for x86_64 emulation
    - Automated setup: `nix run github:ciderale/nixos-utm#nixosCreate .#utm`
    - IP retrieval via ARP cache lookup (MAC-based)
    - Deploy config updates without VM recreation

    **Limitations**: `utmctl` lacks full Apple backend support for some operations

    ### 2. onnimonni/nixos-utm-vm-example ⭐7
    **URL**: https://github.com/onnimonni/nixos-utm-vm-example
    **Purpose**: Practical NixOS VM example for UTM on Apple Silicon
    **Key Features**:
    - Flake-based config management
    - SSH key auth (ed25519)
    - Avahi integration for `.local` hostname access
    - Remote builder pattern for x86_64 Linux builds from ARM host
    - Rosetta emulation support

    ### 3. utmapp/UTM ⭐31.5k
    **URL**: https://github.com/utmapp/UTM
    **Description**: Full-featured VM host for iOS/macOS (QEMU-based)
    **Languages**: Swift (primary), Objective-C, Shell, Python, C

    ## Automation Capabilities

    ### UTM Scripting Options

    #### AppleScript
    ```applescript
    tell application "UTM"
    set vmList to every virtual machine
    set myVM to first virtual machine whose name is "Ubuntu Server"
    start myVM
    suspend myVM
    stop myVM
    end tell
    ```

    #### URL Schemes
    ```
    utm://run?name=MyVM
    utm://pause?name=MyVM
    utm://stop?name=MyVM
    ```

    #### Swift Configuration API
    - Programmatic VM creation/config via `UTMQemuConfiguration`
    - Drive, network, port forwarding, display settings
    - VLAN configuration, multiple network interfaces

    ### NixOS Portable Configuration Patterns

    #### Portable Disk Configuration
    ```nix
    device = "/dev/disk/by-label/nixos" # Works across vda/sda
    ```
    **Benefit**: Same config works for ARM64 (virtualized) & x86_64 (emulated) VMs

    #### Flake Structure
    - Lock dependency versions
    - Modular organization: CLI tools, apps, shell config, dev envs
    - Architecture-specific outputs

    #### Home Manager Integration
    - Declarative user environment
    - `nix run . switch` activates configs

    ## Recommended VM Settings for Apple Silicon

    ### UTM Configuration
    - **Architecture**: ARM64 (aarch64)
    - **Memory**: 8-16GB
    - **CPU**: 4+ cores
    - **Display**: virtio-ramfb-gl (GPU acceleration)
    - **Network**: Shared mode w/ port forwarding (SSH: 22 → 2222)
    - **Boot**: UEFI, RNG Device, Hypervisor enabled

    ### Rosetta2 Support
    Enable x86_64 emulation:
    ```nix
    virtualisation.rosetta.enable = true;
    ```

    ### Network Discovery
    ```nix
    # Avahi for .local hostname access
    services.avahi.enable = true;
    ```

    ## Installation Approaches

    ### 1. Manual Installation Playbook (Detailed)

    #### Prerequisites (macOS Host)
    ```bash
    # Download NixOS ISO (ARM64 minimal with latest kernel)
    # From: https://nixos.org/download.html#nixos-iso
    # Or Hydra: https://hydra.nixos.org/job/nixos/trunk-combined/nixos.iso_minimal.aarch64-linux

    # Verify ISO hash
    shasum -a 256 nixos-minimal-*.iso

    # Create directory for your VM config (optional, for later)
    mkdir -p ~/nix-configs/utm-vm
    cd ~/nix-configs/utm-vm
    ```

    #### Step 1: Create UTM VM
    1. Open UTM → "Create a New Virtual Machine"
    2. Select "Virtualize" (not Emulate)
    3. **Operating System**: Linux
    4. **Boot ISO**: Select downloaded NixOS ISO
    5. **Hardware**:
    - Memory: 8192 MB (8GB) minimum
    - CPU Cores: 4
    6. **Storage**: 100GB (or as needed)
    7. **Shared Directory**: Skip for now (configure post-install)
    8. **Summary**:
    - Name: `nixos-dev`
    - Check "Open VM Settings"
    9. **Settings Tweaks**:
    - System → Enable "UEFI Boot"
    - Display → "virtio-ramfb-gl" for GPU acceleration
    - Network → "Shared Network" + check "Enable Port Forwarding"
    - Guest Port: 22, Host: 2222, Protocol: TCP (for SSH)
    10. Save & Start VM

    #### Step 2: Partition & Format (Inside VM)
    ```bash
    # Boot into NixOS installer, switch to root
    sudo su

    # Identify disk (usually /dev/vda for UTM)
    lsblk

    # Partition with parted
    parted /dev/vda -- mklabel gpt
    parted /dev/vda -- mkpart primary 512MB 100%
    parted /dev/vda -- mkpart ESP fat32 1MB 512MB
    parted /dev/vda -- set 2 esp on

    # Format partitions
    mkfs.ext4 -L nixos /dev/vda1 # Root partition with LABEL
    mkfs.fat -F 32 -n boot /dev/vda2 # Boot partition with LABEL

    # Create swap (optional, adjust size)
    parted /dev/vda -- mkpart swap linux-swap 100% 8GB
    mkswap -L swap /dev/vda3
    swapon /dev/vda3

    # Mount filesystems
    mount /dev/disk/by-label/nixos /mnt
    mkdir -p /mnt/boot
    mount /dev/disk/by-label/boot /mnt/boot
    ```

    #### Step 3: Generate Base Config
    ```bash
    # Generate hardware config
    nixos-generate-config --root /mnt

    # Basic initial config (add vim for editing)
    cat > /mnt/etc/nixos/configuration.nix <<'EOF'
    { config, pkgs, ... }:
    {
    imports = [ ./hardware-configuration.nix ];
    # Bootloader
    boot.loader.systemd-boot.enable = true;
    boot.loader.efi.canTouchEfiVariables = true;
    # Hostname
    networking.hostName = "nixos-utm";
    networking.networkmanager.enable = true;
    # Enable flakes & nix-command
    nix.settings.experimental-features = [ "nix-command" "flakes" ];
    # Timezone
    time.timeZone = "America/New_York"; # Adjust
    # Enable SSH
    services.openssh.enable = true;
    services.openssh.settings.PermitRootLogin = "yes"; # Temporary
    # User account
    users.users.sam = { # Change username
    isNormalUser = true;
    extraGroups = [ "wheel" "networkmanager" ];
    # Temporary password, change after first login
    initialPassword = "changeme";
    };
    # Packages for initial setup
    environment.systemPackages = with pkgs; [
    vim
    git
    curl
    wget
    htop
    ];
    # Rosetta2 support (for x86_64 emulation)
    virtualisation.rosetta.enable = true;
    # Avahi for .local hostname discovery
    services.avahi = {
    enable = true;
    nssmdns4 = true;
    publish = {
    enable = true;
    addresses = true;
    domain = true;
    workstation = true;
    };
    };
    system.stateVersion = "24.11"; # Match ISO version
    }
    EOF

    # Review generated hardware config (uses by-label)
    vim /mnt/etc/nixos/hardware-configuration.nix
    ```

    #### Step 4: Install NixOS
    ```bash
    # Install
    nixos-install

    # Set root password when prompted

    # Shutdown (don't restart yet)
    shutdown -h now
    ```

    #### Step 5: Post-Install UTM Config
    1. In UTM settings, remove installation ISO from CD/DVD drive
    2. Start VM

    #### Step 6: First Boot Setup
    ```bash
    # From macOS host, SSH into VM
    ssh -p 2222 sam@localhost

    # Change user password
    passwd

    # Generate SSH keys for this VM (for git, etc)
    ssh-keygen -t ed25519 -C "sam@nixos-utm"

    # Test internet connectivity
    ping -c 3 nixos.org

    # Verify .local hostname works
    # From another terminal: ping nixos-utm.local
    ```

    #### Step 7: Copy Configs from macOS Host → VM

    ##### Option A: Simple SCP Transfer (Quick Start)
    ```bash
    # From macOS host:

    # Copy SSH keys
    scp -P 2222 ~/.ssh/id_ed25519 sam@localhost:~/.ssh/id_ed25519_host
    scp -P 2222 ~/.ssh/id_ed25519.pub sam@localhost:~/.ssh/id_ed25519_host.pub
    scp -P 2222 ~/.ssh/known_hosts sam@localhost:~/.ssh/

    # Copy git config
    scp -P 2222 ~/.gitconfig sam@localhost:~/

    # Copy SSH config (if you have one)
    scp -P 2222 ~/.ssh/config sam@localhost:~/.ssh/

    # Inside VM: fix permissions
    ssh -p 2222 sam@localhost
    chmod 600 ~/.ssh/id_ed25519_host
    chmod 644 ~/.ssh/id_ed25519_host.pub
    chmod 600 ~/.ssh/config
    ```

    ##### Option B: UTM Shared Directory (Persistent Access)
    1. In UTM VM settings → Sharing
    2. Add shared directory: Select macOS folder (e.g., `~/shared-utm`)
    3. Inside VM:
    ```bash
    # Mount shared directory
    sudo mkdir -p /mnt/shared
    sudo mount -t 9p -o trans=virtio share /mnt/shared

    # Auto-mount on boot: add to /etc/fstab
    # (We'll do this declaratively with NixOS config below)
    ```

    ##### Option C: Home-Manager with Secrets Management (Production)
    See "Advanced: Secrets Management" section below.

    #### Step 8: Initialize Flake-Based Config
    ```bash
    # Inside VM as user
    cd ~
    mkdir -p nix-config
    cd nix-config

    # Create flake.nix
    cat > flake.nix <<'EOF'
    {
    description = "NixOS UTM VM Configuration";
    inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    home-manager = {
    url = "github:nix-community/home-manager";
    inputs.nixpkgs.follows = "nixpkgs";
    };
    };
    outputs = { self, nixpkgs, home-manager, ... }: {
    nixosConfigurations.nixos-utm = nixpkgs.lib.nixosSystem {
    system = "aarch64-linux";
    modules = [
    ./configuration.nix
    home-manager.nixosModules.home-manager
    {
    home-manager.useGlobalPkgs = true;
    home-manager.useUserPackages = true;
    home-manager.users.sam = import ./home.nix;
    }
    ];
    };
    };
    }
    EOF

    # Copy existing config as starting point
    sudo cp /etc/nixos/configuration.nix ./configuration.nix
    sudo cp /etc/nixos/hardware-configuration.nix ./hardware-configuration.nix
    sudo chown sam:users *.nix

    # Create home-manager config
    cat > home.nix <<'EOF'
    { config, pkgs, ... }:
    {
    home.username = "sam";
    home.homeDirectory = "/home/sam";
    home.stateVersion = "24.11";
    # Git configuration
    programs.git = {
    enable = true;
    userName = "Your Name";
    userEmail = "you@example.com";
    extraConfig = {
    init.defaultBranch = "main";
    pull.rebase = true;
    };
    };
    # SSH config
    programs.ssh = {
    enable = true;
    matchBlocks = {
    "github.com" = {
    identityFile = "~/.ssh/id_ed25519";
    };
    };
    };
    # Bash/Shell config
    programs.bash = {
    enable = true;
    shellAliases = {
    ll = "ls -la";
    ".." = "cd ..";
    };
    };
    # Packages for user environment
    home.packages = with pkgs; [
    ripgrep
    fd
    bat
    eza
    ];
    }
    EOF

    # Build and activate
    sudo nixos-rebuild switch --flake .#nixos-utm
    home-manager switch --flake .#nixos-utm
    ```

    #### Step 9: Enable Persistent Shared Directory (Optional)
    ```nix
    # Add to configuration.nix:
    {
    # Auto-mount UTM shared directory
    fileSystems."/mnt/shared" = {
    device = "share";
    fsType = "9p";
    options = [
    "trans=virtio"
    "version=9p2000.L"
    "rw"
    "noauto"
    "x-systemd.automount"
    ];
    };
    }
    ```

    #### Step 10: Version Control Your Config
    ```bash
    # Initialize git repo
    cd ~/nix-config
    git init
    git add .
    git commit -m "init nixos utm config"

    # Push to GitHub (optional)
    gh repo create nixos-utm-config --private --source=. --push
    ```

    #### Step 11: Snapshot VM (Backup)
    In UTM: Right-click VM → "Clone" → Name: `nixos-utm-clean-install`

    ### 2. Automated (nixos-utm)
    ```bash
    export VM_NAME="my-nixos-vm"
    nix run github:ciderale/nixos-utm#nixosCreate .#utm
    ```

    ### 3. Remote Builder Setup
    - Separate ARM64 (virtualized) & x86_64 (emulated) VMs
    - SSH key distribution to root
    - Test: `nix build --impure` validates remote execution

    ---

    ## Advanced: Secrets Management

    For production setups, manage SSH keys & git configs declaratively using secrets management tools.

    ### Option 1: agenix (Recommended for Simplicity)

    #### Setup
    ```bash
    # Add to flake.nix inputs:
    inputs.agenix.url = "github:ryantm/agenix";

    # Add to modules:
    modules = [
    agenix.nixosModules.default
    # or for home-manager:
    # agenix.homeManagerModules.default
    ];
    ```

    #### Create secrets directory
    ```bash
    mkdir -p secrets
    cd secrets

    # Create secrets.nix (defines who can decrypt)
    cat > secrets.nix <<'EOF'
    let
    user1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... user@host";
    vm1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... root@nixos-utm";
    in
    {
    "ssh-key.age".publicKeys = [ user1 vm1 ];
    "gitconfig.age".publicKeys = [ user1 vm1 ];
    }
    EOF

    # Encrypt secrets
    nix run github:ryantm/agenix -- -e ssh-key.age
    # (Paste your private key, save & exit)

    nix run github:ryantm/agenix -- -e gitconfig.age
    # (Paste gitconfig contents)
    ```

    #### Use in home.nix
    ```nix
    { config, pkgs, ... }:
    {
    age.secrets.ssh-key = {
    file = ../secrets/ssh-key.age;
    path = "${config.home.homeDirectory}/.ssh/id_ed25519";
    mode = "600";
    };
    age.secrets.gitconfig = {
    file = ../secrets/gitconfig.age;
    path = "${config.home.homeDirectory}/.gitconfig";
    };
    }
    ```

    ### Option 2: sops-nix (More Features)

    #### Setup
    ```bash
    # Add to flake.nix:
    inputs.sops-nix.url = "github:Mic92/sops-nix";

    # Install sops
    nix-shell -p sops
    ```

    #### Create .sops.yaml
    ```yaml
    keys:
    - &user_sam age1abc...xyz # From: ssh-to-age -i ~/.ssh/id_ed25519
    - &vm age1def...uvw # From: ssh-to-age -i /etc/ssh/ssh_host_ed25519_key.pub

    creation_rules:
    - path_regex: secrets/[^/]+\.yaml$
    key_groups:
    - age:
    - *user_sam
    - *vm
    ```

    #### Encrypt secrets
    ```bash
    # Create secrets file
    sops secrets/default.yaml
    # Edit in $EDITOR:
    # ssh_key: |
    # -----BEGIN OPENSSH PRIVATE KEY-----
    # ...
    # gitconfig: |
    # [user]
    # name = Sam
    ```

    #### Use in configuration
    ```nix
    {
    sops.defaultSopsFile = ../secrets/default.yaml;
    sops.age.sshKeyPaths = [ "/home/sam/.ssh/id_ed25519" ];
    sops.secrets.ssh_key = {
    owner = "sam";
    path = "/home/sam/.ssh/id_ed25519_github";
    };
    }
    ```

    ### Option 3: Git-Crypt (Simple File Encryption)
    ```bash
    # In your nix-config repo:
    nix-shell -p git-crypt

    # Initialize
    git-crypt init

    # Add .gitattributes
    echo "secrets/** filter=git-crypt diff=git-crypt" >> .gitattributes

    # Store keys in secrets/
    mkdir secrets
    cp ~/.ssh/id_ed25519 secrets/
    git add secrets/
    git commit -m "add encrypted secrets"

    # On new machine:
    git-crypt unlock /path/to/key
    ```

    ---

    ## Config Sync Strategies Summary

    | Method | Complexity | Security | Use Case |
    |--------|------------|----------|----------|
    | **SCP Transfer** | Low | Manual | Quick setup, one-time |
    | **Shared Directory** | Low | Host-dependent | Development, frequent changes |
    | **Git (plain)** | Medium | Public only | Public dotfiles |
    | **Git-Crypt** | Medium | Good | Simple encryption |
    | **agenix** | Medium | Excellent | SSH-based, simple |
    | **sops-nix** | High | Excellent | Complex secrets, teams |

    ### Recommended Approach
    1. **Initial setup**: SCP transfer (Step 7, Option A)
    2. **Development**: UTM shared directory (Step 9)
    3. **Production/Portable**: agenix or sops-nix with flake-based config

    ## Best Practices

    ### Configuration Management
    - Use flakes for dependency locking
    - Modular structure: separate hardware, base, VM-specific configs
    - Version control all `.nix` files
    - Label-based disk references (portability)

    ### Performance
    - Virtualization mode (ARM64) > Emulation mode (x86_64)
    - Rosetta2 efficient for x86_64 when needed
    - UTM Apple backend better battery life vs Docker Desktop

    ### Critical Considerations
    - **ACPI Shutdown**: Not supported - avoid host-side restart (recovery: console graphics + installer reattach + fsck)
    - **Network**: Enable wireless networking for bare-metal to reach repos during rebuild
    - **SSH**: Forward port 22 to local port for macOS access

    ## References
    - https://krisztianfekete.org/nixos-on-apple-silicon-with-utm/
    - https://adrianhesketh.com/2024/04/20/setting-up-nixos-remote-builder-m1-mac/
    - https://calcagno.blog/m1dev/
    - https://context7.com/utmapp/utm
    - https://github.com/mitchellh/nixos-config (vm-aarch64-utm configs)
    - https://github.com/a-h/nixos (flake for aarch64 & x86_64)