Running encrypted RAIDZ2 ZFS pool on top of file volumes mounted via s3fs-fuse with MinIO backend

... all on the same machine

0. Requirements

• docker
• s3fs
• zfsutils

1. Install & Run MinIO locally

mkdir -p /data/minio
docker pull minio/minio
docker run -p 9000:9000 minio/minio server /data/minio

2. Mount Bucket into /data/bucket

echo 'minioadmin:minioadmin' > /etc/s3cred
chmod 600 /etc/s3cred
mkdir /data/bucket
s3fs testbucket /data/bucket -o passwd_file=/etc/s3cred,use_path_request_style,url=http://127.0.0.1:9000

3. Create volume files

cd /data/bucket 
dd if=/dev/zero of=volume01 bs=1M count=1024
dd if=/dev/zero of=volume02 bs=1M count=1024
dd if=/dev/zero of=volume03 bs=1M count=1024
dd if=/dev/zero of=volume04 bs=1M count=1024
dd if=/dev/zero of=volume05 bs=1M count=1024
dd if=/dev/zero of=volume06 bs=1M count=1024
dd if=/dev/zero of=volume07 bs=1M count=1024
dd if=/dev/zero of=volume08 bs=1M count=1024
dd if=/dev/zero of=volume09 bs=1M count=1024
dd if=/dev/zero of=volume10 bs=1M count=1024
dd if=/dev/zero of=volume11 bs=1M count=1024
dd if=/dev/zero of=volume12 bs=1M count=1024

4. Create loopback devices

losetup loop11 /data/bucket/volume01
losetup loop12 /data/bucket/volume02
losetup loop13 /data/bucket/volume03
losetup loop14 /data/bucket/volume04
losetup loop15 /data/bucket/volume05
losetup loop16 /data/bucket/volume06
losetup loop17 /data/bucket/volume07
losetup loop18 /data/bucket/volume08
losetup loop19 /data/bucket/volume09
losetup loop20 /data/bucket/volume10
losetup loop21 /data/bucket/volume11
losetup loop22 /data/bucket/volume12

5. Create ZFS pool

mkdir /tank
zpool create -O encryption=aes-256-gcm -O keylocation=prompt -O keyformat=passphrase -O mountpoint=/tank tank raidz2 /dev/loop11 /dev/loop12 /dev/loop13 /dev/loop14 /dev/loop15 /dev/loop16 /dev/loop17 /dev/loop18 /dev/loop19 /dev/loop20 /dev/loop21 /dev/loop22

Check status:

zpool status

Result:

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            loop11  ONLINE       0     0     0
            loop12  ONLINE       0     0     0
            loop13  ONLINE       0     0     0
            loop14  ONLINE       0     0     0
            loop15  ONLINE       0     0     0
            loop16  ONLINE       0     0     0
            loop17  ONLINE       0     0     0
            loop18  ONLINE       0     0     0
            loop19  ONLINE       0     0     0
            loop20  ONLINE       0     0     0
            loop21  ONLINE       0     0     0
            loop22  ONLINE       0     0     0

errors: No known data errors

Check free space:

zfs list

Result:

NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   302K  9.23G      134K  /tank

6. Run benchmarks

dd operation:

  cd /tank
  dd if=/dev/zero of=mybench bs=8k count=100k; rm -f mybench

Result:

102400+0 records in
102400+0 records out
838860800 bytes (839 MB, 800 MiB) copied, 27.6087 s, 30.4 MB/s

bonnie++

bonnie++ -d /tank -s 4000MB -u 1000

Result:

Using uid:1000, gid:65534.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
ubuntu-2gb-he 4000M  115k  99 22.6m  12 14.4m   9  233k  92 49.8m   8 206.8  21
Latency               109ms    1088ms    2036ms   98879us     354ms     259ms
Version  1.98       ------Sequential Create------ --------Random Create--------
ubuntu-2gb-hel1-2   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  86 +++++ +++ 724249376   3     0  86 +++++ +++ 724249376   3
Latency              5041us    1734us     919ms    4731us      82us    1030ms
1.98,1.98,ubuntu-2gb-hel1-2,1,1593535441,4000M,,8192,5,115,99,23158,12,14734,9,233,92,50970,8,206.8,21,16,,,,,16379,86,+++++,+++,498,3,17092,86,+++++,+++,371,3,109ms,1088ms,2036ms,98879us,354ms,259ms,5041us,1734us,919ms,4731us,82us,1030ms