Skip to content

Instantly share code, notes, and snippets.

@TheRainstorm

TheRainstorm/gist:df6e5ceb3c99c037eb13eb951d9aaf59

Last active September 21, 2024 19:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save TheRainstorm/df6e5ceb3c99c037eb13eb951d9aaf59 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save TheRainstorm/df6e5ceb3c99c037eb13eb951d9aaf59 to your computer and use it in GitHub Desktop.
Download ZIP
ZFS raid10 (3TBx2+3x2+2x2) fio Test
Raw
gistfile1.txt
root@ryzen-pve ➜ ~ fio --filename=/mnt/Disk1/test.fio /root/scripts/fio/fio_CrystalDiskMark.ini
Read SEQ1M Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Read SEQ1M Q1T1: (g=1): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Read RND4K Q32T1: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Read RND4K Q1T1: (g=3): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Write SEQ1M Q8T1: (g=4): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Write SEQ1M Q1T1: (g=5): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Write RND4K Q32T1: (g=6): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Write RND4K Q1T1: (g=7): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 8 processes
Read SEQ1M Q8T1: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [_(7),w(1)][55.9%][w=2720KiB/s][w=680 IOPS][eta 04m:01s]
Read SEQ1M Q8T1: (groupid=0, jobs=1): err= 0: pid=86512: Wed Sep 18 00:25:55 2024
read: IOPS=7366, BW=7367MiB/s (7725MB/s)(12.0GiB/1668msec)
slat (usec): min=85, max=357, avg=134.88, stdev=15.53
clat (nsec): min=800, max=2193.0k, avg=948349.38, stdev=86583.35
lat (usec): min=119, max=2550, avg=1083.23, stdev=97.00
clat percentiles (usec):
| 1.00th=[ 799], 5.00th=[ 832], 10.00th=[ 857], 20.00th=[ 881],
| 30.00th=[ 906], 40.00th=[ 922], 50.00th=[ 938], 60.00th=[ 963],
| 70.00th=[ 988], 80.00th=[ 1012], 90.00th=[ 1057], 95.00th=[ 1090],
| 99.00th=[ 1172], 99.50th=[ 1237], 99.90th=[ 1418], 99.95th=[ 1450],
| 99.99th=[ 2040]
bw ( MiB/s): min= 7302, max= 7444, per=100.00%, avg=7369.42, stdev=71.42, samples=3
iops : min= 7302, max= 7444, avg=7369.33, stdev=71.28, samples=3
lat (nsec) : 1000=0.02%
lat (usec) : 250=0.03%, 500=0.05%, 750=0.14%, 1000=75.53%
lat (msec) : 2=24.21%, 4=0.02%
cpu : usr=0.66%, sys=99.22%, ctx=15, majf=4, minf=2061
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Read SEQ1M Q1T1: (groupid=1, jobs=1): err= 0: pid=86551: Wed Sep 18 00:25:55 2024
read: IOPS=7627, BW=7628MiB/s (7998MB/s)(12.0GiB/1611msec)
slat (usec): min=69, max=379, avg=130.29, stdev=14.74
clat (nsec): min=330, max=6270, avg=409.92, stdev=153.13
lat (usec): min=69, max=383, avg=130.70, stdev=14.81
clat percentiles (nsec):
| 1.00th=[ 350], 5.00th=[ 362], 10.00th=[ 362], 20.00th=[ 370],
| 30.00th=[ 370], 40.00th=[ 370], 50.00th=[ 370], 60.00th=[ 382],
| 70.00th=[ 382], 80.00th=[ 390], 90.00th=[ 462], 95.00th=[ 644],
| 99.00th=[ 1080], 99.50th=[ 1304], 99.90th=[ 1720], 99.95th=[ 1880],
| 99.99th=[ 4768]
bw ( MiB/s): min= 7454, max= 7669, per=99.14%, avg=7561.73, stdev=152.35, samples=2
iops : min= 7454, max= 7669, avg=7561.50, stdev=152.03, samples=2
lat (nsec) : 500=90.85%, 750=5.96%, 1000=1.97%
lat (usec) : 2=1.17%, 4=0.02%, 10=0.02%
cpu : usr=0.56%, sys=99.32%, ctx=14, majf=0, minf=269
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Read RND4K Q32T1: (groupid=2, jobs=1): err= 0: pid=86566: Wed Sep 18 00:25:55 2024
read: IOPS=12.8k, BW=49.9MiB/s (52.4MB/s)(2996MiB/60001msec)
slat (nsec): min=1140, max=1294.8k, avg=77361.69, stdev=29892.65
clat (nsec): min=1840, max=4766.8k, avg=2425266.00, stdev=199860.72
lat (usec): min=86, max=4980, avg=2502.63, stdev=203.77
clat percentiles (usec):
| 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2278],
| 30.00th=[ 2343], 40.00th=[ 2376], 50.00th=[ 2442], 60.00th=[ 2474],
| 70.00th=[ 2507], 80.00th=[ 2573], 90.00th=[ 2671], 95.00th=[ 2737],
| 99.00th=[ 2900], 99.50th=[ 3032], 99.90th=[ 3458], 99.95th=[ 3621],
| 99.99th=[ 4015]
bw ( KiB/s): min=48645, max=52256, per=100.00%, avg=51148.20, stdev=605.77, samples=119
iops : min=12161, max=13064, avg=12787.00, stdev=151.43, samples=119
lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=1.85%, 4=98.14%, 10=0.01%
cpu : usr=1.11%, sys=98.74%, ctx=621, majf=0, minf=50
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=767035,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Read RND4K Q1T1: (groupid=3, jobs=1): err= 0: pid=87055: Wed Sep 18 00:25:55 2024
read: IOPS=12.8k, BW=50.0MiB/s (52.4MB/s)(2997MiB/60001msec)
slat (nsec): min=1110, max=1463.1k, avg=77346.18, stdev=34935.75
clat (nsec): min=190, max=119782, avg=383.69, stdev=364.01
lat (nsec): min=1340, max=1464.5k, avg=77729.87, stdev=34990.68
clat percentiles (nsec):
| 1.00th=[ 221], 5.00th=[ 231], 10.00th=[ 241], 20.00th=[ 350],
| 30.00th=[ 362], 40.00th=[ 362], 50.00th=[ 362], 60.00th=[ 370],
| 70.00th=[ 370], 80.00th=[ 382], 90.00th=[ 470], 95.00th=[ 604],
| 99.00th=[ 828], 99.50th=[ 1048], 99.90th=[ 1480], 99.95th=[ 3152],
| 99.99th=[ 5984]
bw ( KiB/s): min=48800, max=55472, per=100.00%, avg=51172.14, stdev=1163.90, samples=119
iops : min=12200, max=13868, avg=12792.99, stdev=290.96, samples=119
lat (nsec) : 250=10.80%, 500=80.75%, 750=6.81%, 1000=1.06%
lat (usec) : 2=0.52%, 4=0.04%, 10=0.02%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%
cpu : usr=1.12%, sys=98.28%, ctx=1090, majf=0, minf=38
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=767350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Write SEQ1M Q8T1: (groupid=4, jobs=1): err= 0: pid=87663: Wed Sep 18 00:25:55 2024
write: IOPS=460, BW=461MiB/s (483MB/s)(12.0GiB/26670msec); 0 zone resets
slat (usec): min=38, max=21815, avg=2168.58, stdev=1527.59
clat (nsec): min=1090, max=81303k, avg=15183521.77, stdev=10567128.63
lat (usec): min=391, max=91964, avg=17352.10, stdev=12067.19
clat percentiles (usec):
| 1.00th=[ 355], 5.00th=[ 766], 10.00th=[ 898], 20.00th=[ 1237],
| 30.00th=[ 6718], 40.00th=[18220], 50.00th=[19530], 60.00th=[20055],
| 70.00th=[20841], 80.00th=[21627], 90.00th=[22414], 95.00th=[26346],
| 99.00th=[46924], 99.50th=[55837], 99.90th=[76022], 99.95th=[78119],
| 99.99th=[81265]
bw ( KiB/s): min=130810, max=5787509, per=97.40%, avg=459511.21, stdev=763896.64, samples=53
iops : min= 127, max= 5651, avg=448.66, stdev=745.89, samples=53
lat (usec) : 2=0.02%, 4=0.01%, 500=4.84%, 750=0.05%, 1000=12.52%
lat (msec) : 2=4.29%, 4=4.96%, 10=6.44%, 20=24.23%, 50=41.85%
lat (msec) : 100=0.81%
cpu : usr=0.66%, sys=4.52%, ctx=10260, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Write SEQ1M Q1T1: (groupid=5, jobs=1): err= 0: pid=87964: Wed Sep 18 00:25:55 2024
write: IOPS=362, BW=362MiB/s (380MB/s)(12.0GiB/33900msec); 0 zone resets
slat (usec): min=1842, max=31081, avg=2756.63, stdev=836.39
clat (nsec): min=450, max=1022.2k, avg=1208.73, stdev=9370.91
lat (usec): min=1843, max=31111, avg=2757.84, stdev=837.19
clat percentiles (nsec):
| 1.00th=[ 490], 5.00th=[ 548], 10.00th=[ 588], 20.00th=[ 684],
| 30.00th=[ 772], 40.00th=[ 884], 50.00th=[ 988], 60.00th=[ 1112],
| 70.00th=[ 1208], 80.00th=[ 1336], 90.00th=[ 1464], 95.00th=[ 1608],
| 99.00th=[ 2064], 99.50th=[ 7264], 99.90th=[29568], 99.95th=[33024],
| 99.99th=[69120]
bw ( KiB/s): min=106496, max=507904, per=100.00%, avg=371214.91, stdev=77629.50, samples=67
iops : min= 104, max= 496, avg=362.45, stdev=75.79, samples=67
lat (nsec) : 500=1.17%, 750=26.25%, 1000=22.66%
lat (usec) : 2=48.85%, 4=0.43%, 10=0.17%, 20=0.18%, 50=0.26%
lat (usec) : 100=0.02%
lat (msec) : 2=0.01%
cpu : usr=0.29%, sys=3.87%, ctx=12313, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Write RND4K Q32T1: (groupid=6, jobs=1): err= 0: pid=88164: Wed Sep 18 00:25:55 2024
write: IOPS=725, BW=2901KiB/s (2970kB/s)(170MiB/60001msec); 0 zone resets
slat (usec): min=612, max=11160, avg=1376.87, stdev=504.23
clat (usec): min=2, max=103667, avg=42721.54, stdev=15435.44
lat (usec): min=1471, max=106321, avg=44098.41, stdev=15931.23
clat percentiles (msec):
| 1.00th=[ 23], 5.00th=[ 24], 10.00th=[ 26], 20.00th=[ 29],
| 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 45],
| 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 71],
| 99.00th=[ 83], 99.50th=[ 84], 99.90th=[ 87], 99.95th=[ 95],
| 99.99th=[ 103]
bw ( KiB/s): min= 1488, max= 5496, per=99.94%, avg=2899.56, stdev=1055.79, samples=119
iops : min= 372, max= 1374, avg=724.84, stdev=263.95, samples=119
lat (usec) : 4=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=67.28%
lat (msec) : 100=32.66%, 250=0.03%
cpu : usr=0.17%, sys=2.79%, ctx=43603, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,43513,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Write RND4K Q1T1: (groupid=7, jobs=1): err= 0: pid=89056: Wed Sep 18 00:25:55 2024
write: IOPS=858, BW=3433KiB/s (3515kB/s)(201MiB/60001msec); 0 zone resets
slat (usec): min=553, max=30183, avg=1163.35, stdev=377.08
clat (nsec): min=290, max=786450, avg=943.04, stdev=3840.54
lat (usec): min=553, max=30206, avg=1164.29, stdev=377.29
clat percentiles (nsec):
| 1.00th=[ 350], 5.00th=[ 382], 10.00th=[ 402], 20.00th=[ 430],
| 30.00th=[ 502], 40.00th=[ 572], 50.00th=[ 652], 60.00th=[ 748],
| 70.00th=[ 908], 80.00th=[ 1096], 90.00th=[ 1384], 95.00th=[ 1592],
| 99.00th=[ 9280], 99.50th=[11840], 99.90th=[18304], 99.95th=[26240],
| 99.99th=[45312]
bw ( KiB/s): min= 2208, max= 5976, per=100.00%, avg=3443.34, stdev=1038.97, samples=119
iops : min= 552, max= 1494, avg=860.83, stdev=259.74, samples=119
lat (nsec) : 500=29.97%, 750=29.28%, 1000=15.60%
lat (usec) : 2=23.09%, 4=0.49%, 10=0.70%, 20=0.78%, 50=0.08%
lat (usec) : 100=0.01%, 250=0.01%, 1000=0.01%
cpu : usr=0.18%, sys=3.39%, ctx=51578, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,51489,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=7367MiB/s (7725MB/s), 7367MiB/s-7367MiB/s (7725MB/s-7725MB/s), io=12.0GiB (12.9GB), run=1668-1668msec
Run status group 1 (all jobs):
READ: bw=7628MiB/s (7998MB/s), 7628MiB/s-7628MiB/s (7998MB/s-7998MB/s), io=12.0GiB (12.9GB), run=1611-1611msec
Run status group 2 (all jobs):
READ: bw=49.9MiB/s (52.4MB/s), 49.9MiB/s-49.9MiB/s (52.4MB/s-52.4MB/s), io=2996MiB (3142MB), run=60001-60001msec
Run status group 3 (all jobs):
READ: bw=50.0MiB/s (52.4MB/s), 50.0MiB/s-50.0MiB/s (52.4MB/s-52.4MB/s), io=2997MiB (3143MB), run=60001-60001msec
Run status group 4 (all jobs):
WRITE: bw=461MiB/s (483MB/s), 461MiB/s-461MiB/s (483MB/s-483MB/s), io=12.0GiB (12.9GB), run=26670-26670msec
Run status group 5 (all jobs):
WRITE: bw=362MiB/s (380MB/s), 362MiB/s-362MiB/s (380MB/s-380MB/s), io=12.0GiB (12.9GB), run=33900-33900msec
Run status group 6 (all jobs):
WRITE: bw=2901KiB/s (2970kB/s), 2901KiB/s-2901KiB/s (2970kB/s-2970kB/s), io=170MiB (178MB), run=60001-60001msec
Run status group 7 (all jobs):
WRITE: bw=3433KiB/s (3515kB/s), 3433KiB/s-3433KiB/s (3515kB/s-3515kB/s), io=201MiB (211MB), run=60001-60001msec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment