Skip to content

Instantly share code, notes, and snippets.

@copecog

copecog/zfs_zoned_pve.md

Created January 2, 2024 20:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save copecog/08996b55827329e31c6d21ccbec45081 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save copecog/08996b55827329e31c6d21ccbec45081 to your computer and use it in GitHub Desktop.
Download ZIP
Testing zfs zone'd datasets with Linux Containers under PVE
Raw
zfs_zoned_pve.md

For debian12base2 CT under PVE

Container

  • Debian 12 (bookworm) with PVE/PBS APT sources added, primarily for matching ZFS userspace distrib, but also to optionally install any PVE/PBS components

APT Sources Host and CT

/etc/apt/sources.list

deb https://mirrors.xmission.com/debian bookworm main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-backports main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-backports main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-backports-sloppy main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-backports-sloppy main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-proposed-updates main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-proposed-updates main contrib non-free non-free-firmware

deb https://mirrors.xmission.com/debian bookworm-updates main contrib non-free non-free-firmware
deb-src https://mirrors.xmission.com/debian bookworm-updates main contrib non-free non-free-firmware

# security updates
deb https://security.debian.org bookworm-security main contrib non-free non-free-firmware

/etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

/etc/apt/sources.list.d/pve-enterprise.list

#deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise

/etc/apt/sources.list.d/pve-no-subscription.list

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

/etc/apt/sources.list.d/pbs-enterprise.list

#deb https://enterprise.proxmox.com/debian/pbs bookworm pbs-enterprise

/etc/apt/sources.list.d/pbs-no-subscription.list

deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription

LXC /dev/zfs Mapping

root@soldier10:~# cat /etc/pve/nodes/soldier10/lxc/103.conf
...
lxc.cgroup2.devices.allow: c 10 249 rwm
lxc.mount.entry: /dev/zfs dev/zfs none bind,create=file

ZFS Dataset Settings on Host

# Create datasets to namespace into containers
zfs create -o mountpoint=none rpool/ct
zfs create rpool/ct/103

# Find init process of container (Started after /dev/zfs mapping)
ps faux
...
root      123684  0.0  0.0   5024  2816 ?        Ss    2023   2:07 /usr/bin/lxc-start -F -n 103
100000    123700  0.0  0.0 167728  9472 ?        Ss    2023   0:11  \_ /sbin/init
...

# Verify user namespace exists (unprivileged container)
root@soldier10:~# lsns | grep 123700
...
4026532835 user       22 123700 100000    ├─/sbin/init
...

# Set namespace for dataset
zfs set zoned=on rpool/ct/103
zfs zone /proc/123700/ns/user rpool/ct/103

ZFS Settings in CT

  • Set APT Sources
  • apt install zfsutils-linux to match host

View in CT

root@debian12base2:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool              63.9G   386G    96K  /rpool
rpool/ct            480K   386G    96K  none
rpool/ct/103        192K   386G    96K  none

root@debian12base2:~# zfs create -o mountpoint=/tank rpool/ct/103/tank
root@debian12base2:~# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool              63.9G   386G    96K  /rpool
rpool/ct            480K   386G    96K  none
rpool/ct/103        192K   386G    96K  none
rpool/ct/103/tank    96K   386G    96K  /tank

root@debian12base2:~# mount | grep tank
rpool/ct/103/tank on /tank type zfs (rw,noatime,xattr,noacl,casesensitive)

Next

  • Lookup environment variables for PVE hookscript and see if I can automate zfs zone and zfs unzone
  • Try ZFS Docker driver in CT
@copecog
Copy link
Author

copecog commented Mar 21, 2024

If your interested I've created a quick hook script to automate zfs zone.

Thanks! I'll give this a try!

@tcpluess
Copy link

tcpluess commented Sep 2, 2024

How is this hook script to be used?
can the ZFS zoning in Proxmox be automated somehow?

@Malzmeier
Copy link

Malzmeier commented Sep 2, 2024 •
edited
Loading

It's a proxmox hookscript. Documentation is located here: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_hookscripts
You have to place it in /var/lib/vz/snippets and add it to the conf of your lxc at /etc/pve/lxc by adding (in my configuration)
hookscript: local:snippets/hook121.sh
I'm using this without any problems to automate ZFS zoning on my proxmox.

This is my hookscript (hook121.sh):

#!/bin/bash

vmId="$1"
runPhase="$2"
echo "Running $runPhase on VM=$vmId"

case "$runPhase" in
    post-start)
        pid=$( lxc-info -n $1 -p | awk '/PID:/{print $NF}' )
        zfs zone /proc/$pid/ns/user sata/ct/urbackup
        echo "zfs zone"
        ;;
    pre-stop)
        pid=$( lxc-info -n $1 -p | awk '/PID:/{print $NF}' )
        zfs unzone /proc/$pid/ns/user sata/ct/urbackup
        echo "zfs unzone"
        ;;
esac

@tcpluess
Copy link

tcpluess commented Sep 10, 2024

very good. This works indeed very well.
However, I can see that I still have access to the entire pool from the LXC, for example, I can do zpool scrub from the LXC. Also what is not so nice is that the mountpoint for newly created datasets is not configured automatically.

Is it somehow possible to hide the pool from "tpool status" and/or disable access to the pool itsel for the LXC, and is it further possible to automatically set the mountpoints for newly created datasets, like it is the normal behaviour when one is working directly on the host?

@antst
Copy link

antst commented Oct 28, 2024

It works for unprivileged containers, but does not work for unprivileged ones. (I need to look into Incus code, how they handle this)

And pre-stop is too early to unzone, ideally it must be post-stop. (otherwise poweroff or reboot in container can bring a lot of fun, depending on situation). So, it has to go deeper to the proxmox code. But Proxmox devs refusing to even consider zfs zoning for containers.

@tcpluess
Copy link

tcpluess commented Dec 31, 2024

Hi antst,
do you think it is somehow possible to make this work with a privileged container?
I just tried it and it fails. The error shown is: "Failed to initialize the libzfs library." but I cannot figure out why.

@DeViLRuNNeR-dev
Copy link

DeViLRuNNeR-dev commented Feb 13, 2025 •
edited
Loading

Amazing work and weird that this isn't more documented or used as the potential of ZFS zoning is particularly interesting in HA setups (my use case).
I did have to chmod 0 /usr/sbin/reboot in the container as rebooting the container or LXC without the host OS 'knowing' - thus not remapping the new namespace - screws things up.

@DeViLRuNNeR-dev
Copy link

DeViLRuNNeR-dev commented Feb 13, 2025 •
edited
Loading

If anybody comes across this and need some inspiration - below the hookscript I tinkered together (doing some other 'hacky stuff').

Would be interesting to hear how anybody else is handling when the container itself is rebooted and thus 'breaks' the ZFS namespace.

#!/bin/sh

vmId="$1"
runPhase="$2"
#--change these variables--#
ZFSdataset="SATA-ZFS/zoned"
hosts_line_patternkeyword="DaC"
#--------------------------#
pid=$(lxc-info -n $vmId -p | awk '/PID:/{print $NF}')
hosts_line=$(awk "/$hosts_line_patternkeyword/" /etc/hosts)
IMAGE=$(pct config $vmId | awk -F'[: =]' '/rootfs: /{print"/"$3"/"$4}' | awk -F ',' '{print$1}')

echo "Running $runPhase on VM=$vmId"

case "$runPhase" in
    post-start)
	# Set dataset zoning
	zfs set zoned=on $ZFSdataset
	echo "executed on Host("$(hostname)"):zfs set zoned=on "$ZFSdataset
	# Set namespace for dataset
        zfs zone /proc/$pid/ns/user $ZFSdataset
        echo "executed on Host("$(hostname)"):zfs zone /proc/"$pid"/ns/user "$ZFSdataset
	# Copy specific host entry from host hosts to container hosts file
	sed -i "/$hosts_line_patternkeyword/d" $IMAGE"/etc/hosts" && awk "/$hosts_line_patternkeyword/" /etc/hosts >> $IMAGE"/etc/hosts"
	echo "Added line "$hosts_line" from /etc/hosts on Host("$(hostname)") to "$IMAGE"/etc/hosts"
        ;;
#    pre-stop)
    post-stop)
	# UnSet namespace for dataset
        zfs unzone /proc/$pid/ns/user $ZFSdataset
        echo "executed on Host("$(hostname)"):zfs unzone /proc/"$pid"/ns/user "$ZFSdataset
	# UnSet dataset zoning
	zfs set zoned=off $ZFSdataset
	echo "executed on Host("$(hostname)"):zfs set zoned=off "$ZFSdataset
        ;;
esac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment