Skip to content

Instantly share code, notes, and snippets.

@copecog

copecog/zfs_zoned_pve.md

Created January 2, 2024 20:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save copecog/08996b55827329e31c6d21ccbec45081 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save copecog/08996b55827329e31c6d21ccbec45081 to your computer and use it in GitHub Desktop.
Download ZIP
Testing zfs zone'd datasets with Linux Containers under PVE
Raw
zfs_zoned_pve.md

For debian12base2 CT under PVE

Container

  • Debian 12 (bookworm) with PVE/PBS APT sources added, primarily for matching ZFS userspace distrib, but also to optionally install any PVE/PBS components

APT Sources Host and CT

/etc/apt/sources.list

deb https://mirrors.xmission.com/debian bookworm main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-backports main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-backports main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-backports-sloppy main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-backports-sloppy main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-proposed-updates main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-proposed-updates main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-updates main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-updates main contrib non-free non-free-firmware

# security updates
deb https://security.debian.org bookworm-security main contrib non-free non-free-firmware

/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

/etc/apt/sources.list.d/pve-enterprise.list

#deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise

/etc/apt/sources.list.d/pve-no-subscription.list

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

/etc/apt/sources.list.d/pbs-enterprise.list

#deb https://enterprise.proxmox.com/debian/pbs bookworm pbs-enterprise

/etc/apt/sources.list.d/pbs-no-subscription.list

deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription

LXC /dev/zfs Mapping

root@soldier10:~# cat /etc/pve/nodes/soldier10/lxc/103.conf
...
lxc.cgroup2.devices.allow: c 10 249 rwm
lxc.mount.entry: /dev/zfs dev/zfs none bind,create=file

ZFS Dataset Settings on Host

# Create datasets to namespace into containers
zfs create -o mountpoint=none rpool/ct
zfs create rpool/ct/103

# Find init process of container (Started after /dev/zfs mapping)
ps faux
...
root      123684  0.0  0.0   5024  2816 ?        Ss    2023   2:07 /usr/bin/lxc-start -F -n 103
100000    123700  0.0  0.0 167728  9472 ?        Ss    2023   0:11  \_ /sbin/init
...

# Verify user namespace exists (unprivileged container)
root@soldier10:~# lsns | grep 123700
...
4026532835 user       22 123700 100000    ├─/sbin/init
...

# Set namespace for dataset
zfs set zoned=on rpool/ct/103
zfs zone /proc/123700/ns/user rpool/ct/103

ZFS Settings in CT

  • Set APT Sources
  • apt install zfsutils-linux to match host

View in CT

root@debian12base2:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool              63.9G   386G    96K  /rpool
rpool/ct            480K   386G    96K  none
rpool/ct/103        192K   386G    96K  none

root@debian12base2:~# zfs create -o mountpoint=/tank rpool/ct/103/tank
root@debian12base2:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool              63.9G   386G    96K  /rpool
rpool/ct            480K   386G    96K  none
rpool/ct/103        192K   386G    96K  none
rpool/ct/103/tank    96K   386G    96K  /tank

root@debian12base2:~# mount | grep tank
rpool/ct/103/tank on /tank type zfs (rw,noatime,xattr,noacl,casesensitive)

Next

  • Lookup environment variables for PVE hookscript and see if I can automate zfs zone and zfs unzone
  • Try ZFS Docker driver in CT
@DeViLRuNNeR-dev
Copy link

DeViLRuNNeR-dev commented Feb 13, 2025 •
edited
Loading

Amazing work and weird that this isn't more documented or used as the potential of ZFS zoning is particularly interesting in HA setups (my use case).
I did have to chmod 0 /usr/sbin/reboot in the container as rebooting the container or LXC without the host OS 'knowing' - thus not remapping the new namespace - screws things up.

@DeViLRuNNeR-dev
Copy link

DeViLRuNNeR-dev commented Feb 13, 2025 •
edited
Loading

If anybody comes across this and need some inspiration - below the hookscript I tinkered together (doing some other 'hacky stuff').

Would be interesting to hear how anybody else is handling when the container itself is rebooted and thus 'breaks' the ZFS namespace.

#!/bin/sh

vmId="$1"
runPhase="$2"
#--change these variables--#
ZFSdataset="SATA-ZFS/zoned"
hosts_line_patternkeyword="DaC"
#--------------------------#
pid=$(lxc-info -n $vmId -p | awk '/PID:/{print $NF}')
hosts_line=$(awk "/$hosts_line_patternkeyword/" /etc/hosts)
IMAGE=$(pct config $vmId | awk -F'[: =]' '/rootfs: /{print"/"$3"/"$4}' | awk -F ',' '{print$1}')

echo "Running $runPhase on VM=$vmId"

case "$runPhase" in
    post-start)
	# Set dataset zoning
	zfs set zoned=on $ZFSdataset
	echo "executed on Host("$(hostname)"):zfs set zoned=on "$ZFSdataset
	# Set namespace for dataset
        zfs zone /proc/$pid/ns/user $ZFSdataset
        echo "executed on Host("$(hostname)"):zfs zone /proc/"$pid"/ns/user "$ZFSdataset
	# Copy specific host entry from host hosts to container hosts file
	sed -i "/$hosts_line_patternkeyword/d" $IMAGE"/etc/hosts" && awk "/$hosts_line_patternkeyword/" /etc/hosts >> $IMAGE"/etc/hosts"
	echo "Added line "$hosts_line" from /etc/hosts on Host("$(hostname)") to "$IMAGE"/etc/hosts"
        ;;
#    pre-stop)
    post-stop)
	# UnSet namespace for dataset
        zfs unzone /proc/$pid/ns/user $ZFSdataset
        echo "executed on Host("$(hostname)"):zfs unzone /proc/"$pid"/ns/user "$ZFSdataset
	# UnSet dataset zoning
	zfs set zoned=off $ZFSdataset
	echo "executed on Host("$(hostname)"):zfs set zoned=off "$ZFSdataset
        ;;
esac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment