Skip to content

Instantly share code, notes, and snippets.

@TheRainstorm

TheRainstorm/acsalab snode2 zeus nfs

Last active March 13, 2025 08:50
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save TheRainstorm/426a5dbe1f08dc99384d786cf46a8796 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save TheRainstorm/426a5dbe1f08dc99384d786cf46a8796 to your computer and use it in GitHub Desktop.
Download ZIP
fio
Raw
acsalab snode2 zeus nfs
Read SEQ1M Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Read RND4K Q32T1: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Read RND4K Q1T1: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Write SEQ1M Q8T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Write RND4K Q32T1: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Write RND4K Q1T1: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.16
Starting 6 processes
Read SEQ1M Q8T1: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [_(5),w(1)][58.8%][w=14.7MiB/s][w=3752 IOPS][eta 02m:27s]
Read SEQ1M Q8T1: (groupid=0, jobs=1): err= 0: pid=1572769: Thu Mar 13 15:08:11 2025
read: IOPS=5887, BW=5888MiB/s (6174MB/s)(12.0GiB/2087msec)
slat (usec): min=50, max=570, avg=112.73, stdev=42.25
clat (usec): min=647, max=5806, avg=1127.19, stdev=143.21
lat (usec): min=725, max=5960, avg=1240.09, stdev=145.83
clat percentiles (usec):
| 1.00th=[ 889], 5.00th=[ 979], 10.00th=[ 1037], 20.00th=[ 1057],
| 30.00th=[ 1074], 40.00th=[ 1106], 50.00th=[ 1123], 60.00th=[ 1139],
| 70.00th=[ 1156], 80.00th=[ 1172], 90.00th=[ 1221], 95.00th=[ 1303],
| 99.00th=[ 1516], 99.50th=[ 1631], 99.90th=[ 2057], 99.95th=[ 4293],
| 99.99th=[ 5276]
bw ( MiB/s): min= 4284, max= 4894, per=76.63%, avg=4511.97, stdev=333.57, samples=3
iops : min= 4284, max= 4894, avg=4511.33, stdev=333.35, samples=3
lat (usec) : 750=0.07%, 1000=6.14%
lat (msec) : 2=93.67%, 4=0.05%, 10=0.07%
cpu : usr=0.96%, sys=76.46%, ctx=7912, majf=0, minf=2060
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Read RND4K Q32T1: (groupid=1, jobs=1): err= 0: pid=1572774: Thu Mar 13 15:08:11 2025
read: IOPS=152k, BW=593MiB/s (622MB/s)(12.0GiB/20728msec)
slat (nsec): min=2513, max=99219, avg=5187.33, stdev=1347.87
clat (usec): min=104, max=8901, avg=205.20, stdev=46.09
lat (usec): min=111, max=8905, avg=210.49, stdev=46.14
clat percentiles (usec):
| 1.00th=[ 182], 5.00th=[ 190], 10.00th=[ 192], 20.00th=[ 196],
| 30.00th=[ 198], 40.00th=[ 200], 50.00th=[ 202], 60.00th=[ 204],
| 70.00th=[ 208], 80.00th=[ 212], 90.00th=[ 219], 95.00th=[ 227],
| 99.00th=[ 265], 99.50th=[ 289], 99.90th=[ 375], 99.95th=[ 453],
| 99.99th=[ 2376]
bw ( KiB/s): min=480040, max=567504, per=85.28%, avg=517696.88, stdev=13327.04, samples=41
iops : min=120010, max=141876, avg=129423.73, stdev=3331.81, samples=41
lat (usec) : 250=98.41%, 500=1.56%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=23.73%, sys=75.39%, ctx=11132, majf=0, minf=77
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=3145728,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Read RND4K Q1T1: (groupid=2, jobs=1): err= 0: pid=1572846: Thu Mar 13 15:08:11 2025
read: IOPS=14.0k, BW=54.7MiB/s (57.4MB/s)(3283MiB/60001msec)
slat (nsec): min=5004, max=133857, avg=8869.72, stdev=976.52
clat (usec): min=9, max=6861, avg=61.09, stdev=35.61
lat (usec): min=54, max=6872, avg=70.24, stdev=35.68
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 56], 10.00th=[ 57], 20.00th=[ 58],
| 30.00th=[ 59], 40.00th=[ 60], 50.00th=[ 60], 60.00th=[ 61],
| 70.00th=[ 62], 80.00th=[ 63], 90.00th=[ 66], 95.00th=[ 69],
| 99.00th=[ 81], 99.50th=[ 93], 99.90th=[ 137], 99.95th=[ 212],
| 99.99th=[ 1663]
bw ( KiB/s): min=42422, max=57185, per=88.23%, avg=49428.00, stdev=1727.10, samples=119
iops : min=10605, max=14296, avg=12356.65, stdev=431.82, samples=119
lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=99.62%, 250=0.33%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=7.43%, sys=18.41%, ctx=840411, majf=0, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=840387,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Write SEQ1M Q8T1: (groupid=3, jobs=1): err= 0: pid=1573011: Thu Mar 13 15:08:11 2025
write: IOPS=2444, BW=2444MiB/s (2563MB/s)(12.0GiB/5027msec); 0 zone resets
slat (usec): min=60, max=1354, avg=183.92, stdev=82.29
clat (usec): min=700, max=20053, avg=3083.03, stdev=2524.62
lat (usec): min=842, max=20259, avg=3267.34, stdev=2542.44
clat percentiles (usec):
| 1.00th=[ 898], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1270],
| 30.00th=[ 1401], 40.00th=[ 1516], 50.00th=[ 1696], 60.00th=[ 2180],
| 70.00th=[ 3589], 80.00th=[ 5669], 90.00th=[ 6587], 95.00th=[ 7701],
| 99.00th=[11863], 99.50th=[12780], 99.90th=[15664], 99.95th=[16188],
| 99.99th=[17957]
bw ( MiB/s): min= 1045, max= 5424, per=97.57%, avg=2385.11, stdev=1266.22, samples=9
iops : min= 1045, max= 5424, avg=2384.67, stdev=1266.17, samples=9
lat (usec) : 750=0.03%, 1000=3.31%
lat (msec) : 2=53.73%, 4=14.54%, 10=26.51%, 20=1.87%, 50=0.01%
cpu : usr=7.50%, sys=40.13%, ctx=4263, majf=0, minf=4130
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Write RND4K Q32T1: (groupid=4, jobs=1): err= 0: pid=1573023: Thu Mar 13 15:08:11 2025
write: IOPS=5608, BW=21.9MiB/s (22.0MB/s)(1315MiB/60005msec); 0 zone resets
slat (usec): min=4, max=136, avg=10.94, stdev= 4.18
clat (usec): min=290, max=45505, avg=5692.73, stdev=4399.21
lat (usec): min=300, max=45514, avg=5703.96, stdev=4400.26
clat percentiles (usec):
| 1.00th=[ 445], 5.00th=[ 799], 10.00th=[ 1270], 20.00th=[ 2114],
| 30.00th=[ 3195], 40.00th=[ 4047], 50.00th=[ 4948], 60.00th=[ 5669],
| 70.00th=[ 6390], 80.00th=[ 7504], 90.00th=[12518], 95.00th=[14877],
| 99.00th=[20317], 99.50th=[24773], 99.90th=[29230], 99.95th=[30278],
| 99.99th=[38011]
bw ( KiB/s): min= 5606, max=32584, per=68.75%, avg=15423.87, stdev=6344.69, samples=119
iops : min= 1401, max= 8146, avg=3855.61, stdev=1586.21, samples=119
lat (usec) : 500=1.92%, 750=2.50%, 1000=2.79%
lat (msec) : 2=11.42%, 4=20.93%, 10=47.51%, 20=11.87%, 50=1.07%
cpu : usr=3.55%, sys=9.34%, ctx=294210, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,336540,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Write RND4K Q1T1: (groupid=5, jobs=1): err= 0: pid=1573299: Thu Mar 13 15:08:11 2025
write: IOPS=3641, BW=14.2MiB/s (14.9MB/s)(854MiB/60005msec); 0 zone resets
slat (usec): min=4, max=189, avg=10.25, stdev= 3.88
clat (usec): min=11, max=30710, avg=262.54, stdev=1339.93
lat (usec): min=67, max=30738, avg=273.10, stdev=1341.65
clat percentiles (usec):
| 1.00th=[ 68], 5.00th=[ 71], 10.00th=[ 72], 20.00th=[ 75],
| 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 85], 60.00th=[ 94],
| 70.00th=[ 105], 80.00th=[ 122], 90.00th=[ 196], 95.00th=[ 297],
| 99.00th=[ 8717], 99.50th=[13173], 99.90th=[14746], 99.95th=[16319],
| 99.99th=[25822]
bw ( KiB/s): min= 272, max=33488, per=99.99%, avg=14564.14, stdev=8824.63, samples=120
iops : min= 68, max= 8372, avg=3640.99, stdev=2206.17, samples=120
lat (usec) : 20=0.01%, 50=0.01%, 100=65.82%, 250=28.05%, 500=3.73%
lat (usec) : 750=0.24%, 1000=0.17%
lat (msec) : 2=0.53%, 4=0.25%, 10=0.44%, 20=0.73%, 50=0.03%
cpu : usr=2.19%, sys=6.00%, ctx=218529, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,218513,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=5888MiB/s (6174MB/s), 5888MiB/s-5888MiB/s (6174MB/s-6174MB/s), io=12.0GiB (12.9GB), run=2087-2087msec
Run status group 1 (all jobs):
READ: bw=593MiB/s (622MB/s), 593MiB/s-593MiB/s (622MB/s-622MB/s), io=12.0GiB (12.9GB), run=20728-20728msec
Run status group 2 (all jobs):
READ: bw=54.7MiB/s (57.4MB/s), 54.7MiB/s-54.7MiB/s (57.4MB/s-57.4MB/s), io=3283MiB (3442MB), run=60001-60001msec
Run status group 3 (all jobs):
WRITE: bw=2444MiB/s (2563MB/s), 2444MiB/s-2444MiB/s (2563MB/s-2563MB/s), io=12.0GiB (12.9GB), run=5027-5027msec
Run status group 4 (all jobs):
WRITE: bw=21.9MiB/s (22.0MB/s), 21.9MiB/s-21.9MiB/s (22.0MB/s-22.0MB/s), io=1315MiB (1378MB), run=60005-60005msec
Run status group 5 (all jobs):
WRITE: bw=14.2MiB/s (14.9MB/s), 14.2MiB/s-14.2MiB/s (14.9MB/s-14.9MB/s), io=854MiB (895MB), run=60005-60005msec
Raw
acsalab zeus zfs raidz2(18T*8) * 3
root@zeus:~# fio --loops=3 --size=4g --filename=/rpool/test/fio.img --runtime=60 --ioengine=libaio --direct=1 \
--name=SeqRead --bs=1m --iodepth=8 --rw=read --stonewall \
--name=SeqWrite --bs=1m --iodepth=8 --rw=write --stonewall \
--name=4KQD32read --bs=4k --iodepth=32 --rw=randread --stonewall \
--name=4KQD32write --bs=4k --iodepth=32 --rw=randwrite --stonewall \
--name=4Kread --bs=4k --rw=randread --stonewall \
--name=4Kwrite --bs=4k --rw=randwrite --stonewall
SeqRead: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
SeqWrite: (g=1): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
4KQD32read: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
4KQD32write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
4Kread: (g=4): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
4Kwrite: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 6 processes
Jobs: 1 (f=1): [_(5),w(1)][100.0%][w=264MiB/s][w=67.5k IOPS][eta 00m:00s]
SeqRead: (groupid=0, jobs=1): err= 0: pid=2444307: Thu Mar 13 16:39:32 2025
read: IOPS=1516, BW=1517MiB/s (1590MB/s)(12.0GiB/8102msec)
slat (usec): min=37, max=650, avg=45.40, stdev=17.87
clat (usec): min=1217, max=8695, avg=5226.67, stdev=594.47
lat (usec): min=1262, max=8749, avg=5272.07, stdev=593.34
clat percentiles (usec):
| 1.00th=[ 3949], 5.00th=[ 4359], 10.00th=[ 4490], 20.00th=[ 4752],
| 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5211], 60.00th=[ 5342],
| 70.00th=[ 5473], 80.00th=[ 5669], 90.00th=[ 5997], 95.00th=[ 6194],
| 99.00th=[ 6783], 99.50th=[ 7046], 99.90th=[ 7635], 99.95th=[ 8225],
| 99.99th=[ 8586]
bw ( MiB/s): min= 1508, max= 1529, per=100.00%, avg=1516.71, stdev= 5.73, samples=15
iops : min= 1508, max= 1529, avg=1516.67, stdev= 5.77, samples=15
lat (msec) : 2=0.03%, 4=1.33%, 10=98.63%
cpu : usr=0.35%, sys=8.26%, ctx=12147, majf=0, minf=2060
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
SeqWrite: (groupid=1, jobs=1): err= 0: pid=2444449: Thu Mar 13 16:39:32 2025
write: IOPS=3257, BW=3258MiB/s (3416MB/s)(12.0GiB/3772msec); 0 zone resets
slat (usec): min=43, max=667, avg=86.96, stdev=33.11
clat (usec): min=559, max=6420, avg=2366.40, stdev=280.02
lat (usec): min=640, max=6595, avg=2453.36, stdev=283.66
clat percentiles (usec):
| 1.00th=[ 1795], 5.00th=[ 2147], 10.00th=[ 2180], 20.00th=[ 2245],
| 30.00th=[ 2278], 40.00th=[ 2278], 50.00th=[ 2311], 60.00th=[ 2343],
| 70.00th=[ 2409], 80.00th=[ 2474], 90.00th=[ 2573], 95.00th=[ 2704],
| 99.00th=[ 3458], 99.50th=[ 3720], 99.90th=[ 5604], 99.95th=[ 5932],
| 99.99th=[ 6390]
bw ( MiB/s): min= 3212, max= 3304, per=100.00%, avg=3263.34, stdev=40.58, samples=7
iops : min= 3212, max= 3304, avg=3263.14, stdev=40.53, samples=7
lat (usec) : 750=0.07%, 1000=0.10%
lat (msec) : 2=2.12%, 4=97.42%, 10=0.28%
cpu : usr=14.61%, sys=14.43%, ctx=12216, majf=0, minf=543
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
4KQD32read: (groupid=2, jobs=1): err= 0: pid=2444475: Thu Mar 13 16:39:32 2025
read: IOPS=255k, BW=995MiB/s (1044MB/s)(12.0GiB/12344msec)
slat (nsec): min=1869, max=214910, avg=2956.30, stdev=1614.65
clat (usec): min=10, max=770, avg=122.24, stdev=49.97
lat (usec): min=12, max=773, avg=125.19, stdev=50.12
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 75], 20.00th=[ 84],
| 30.00th=[ 92], 40.00th=[ 100], 50.00th=[ 110], 60.00th=[ 120],
| 70.00th=[ 135], 80.00th=[ 157], 90.00th=[ 188], 95.00th=[ 221],
| 99.00th=[ 293], 99.50th=[ 326], 99.90th=[ 396], 99.95th=[ 429],
| 99.99th=[ 498]
bw ( KiB/s): min=961647, max=1063799, per=100.00%, avg=1020728.00, stdev=37156.25, samples=24
iops : min=240411, max=265949, avg=255181.88, stdev=9289.02, samples=24
lat (usec) : 20=0.01%, 50=0.20%, 100=39.59%, 250=57.62%, 500=2.59%
lat (usec) : 750=0.01%, 1000=0.01%
cpu : usr=22.22%, sys=76.80%, ctx=24714, majf=0, minf=47
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=3145728,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
4KQD32write: (groupid=3, jobs=1): err= 0: pid=2444743: Thu Mar 13 16:39:32 2025
write: IOPS=191k, BW=748MiB/s (784MB/s)(12.0GiB/16429msec); 0 zone resets
slat (nsec): min=1853, max=234207, avg=3699.67, stdev=2496.63
clat (nsec): min=375, max=3906.2k, avg=162892.47, stdev=28686.17
lat (usec): min=11, max=3917, avg=166.59, stdev=29.06
clat percentiles (usec):
| 1.00th=[ 124], 5.00th=[ 130], 10.00th=[ 135], 20.00th=[ 145],
| 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 167],
| 70.00th=[ 172], 80.00th=[ 176], 90.00th=[ 182], 95.00th=[ 188],
| 99.00th=[ 251], 99.50th=[ 310], 99.90th=[ 433], 99.95th=[ 457],
| 99.99th=[ 562]
bw ( KiB/s): min=693450, max=798128, per=100.00%, avg=766719.44, stdev=27491.68, samples=32
iops : min=173362, max=199530, avg=191679.62, stdev=6872.92, samples=32
lat (nsec) : 500=0.01%
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=98.99%, 500=0.99%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=11.75%, sys=65.22%, ctx=1818704, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3145728,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
4Kread: (groupid=4, jobs=1): err= 0: pid=2444931: Thu Mar 13 16:39:32 2025
read: IOPS=13.3k, BW=51.8MiB/s (54.3MB/s)(3107MiB/60001msec)
slat (nsec): min=1940, max=669094, avg=2574.10, stdev=2223.74
clat (nsec): min=1433, max=8160.1k, avg=72385.69, stdev=30538.79
lat (usec): min=11, max=8165, avg=74.96, stdev=30.74
clat percentiles (usec):
| 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 49],
| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 82],
| 70.00th=[ 83], 80.00th=[ 84], 90.00th=[ 106], 95.00th=[ 145],
| 99.00th=[ 149], 99.50th=[ 149], 99.90th=[ 165], 99.95th=[ 215],
| 99.99th=[ 594]
bw ( KiB/s): min=48846, max=54212, per=100.00%, avg=53050.49, stdev=1111.63, samples=119
iops : min=12211, max=13553, avg=13262.49, stdev=277.89, samples=119
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=27.01%
lat (usec) : 100=62.16%, 250=10.78%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.94%, sys=5.83%, ctx=795555, majf=1, minf=75
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=795398,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
4Kwrite: (groupid=5, jobs=1): err= 0: pid=2445818: Thu Mar 13 16:39:32 2025
write: IOPS=73.0k, BW=285MiB/s (299MB/s)(12.0GiB/43101msec); 0 zone resets
slat (usec): min=2, max=488, avg= 2.39, stdev= 1.02
clat (nsec): min=482, max=7049.0k, avg=10923.85, stdev=14938.01
lat (usec): min=10, max=7054, avg=13.31, stdev=15.04
clat percentiles (usec):
| 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10],
| 30.00th=[ 11], 40.00th=[ 11], 50.00th=[ 11], 60.00th=[ 11],
| 70.00th=[ 11], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 14],
| 99.00th=[ 23], 99.50th=[ 25], 99.90th=[ 79], 99.95th=[ 108],
| 99.99th=[ 182]
bw ( KiB/s): min=254712, max=312768, per=100.00%, avg=292132.65, stdev=13599.29, samples=86
iops : min=63678, max=78192, avg=73033.21, stdev=3399.75, samples=86
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=31.17%, 20=67.27%, 50=1.39%
lat (usec) : 100=0.10%, 250=0.06%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=6.87%, sys=23.78%, ctx=3146924, majf=1, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3145728,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=1517MiB/s (1590MB/s), 1517MiB/s-1517MiB/s (1590MB/s-1590MB/s), io=12.0GiB (12.9GB), run=8102-8102msec
Run status group 1 (all jobs):
WRITE: bw=3258MiB/s (3416MB/s), 3258MiB/s-3258MiB/s (3416MB/s-3416MB/s), io=12.0GiB (12.9GB), run=3772-3772msec
Run status group 2 (all jobs):
READ: bw=995MiB/s (1044MB/s), 995MiB/s-995MiB/s (1044MB/s-1044MB/s), io=12.0GiB (12.9GB), run=12344-12344msec
Run status group 3 (all jobs):
WRITE: bw=748MiB/s (784MB/s), 748MiB/s-748MiB/s (784MB/s-784MB/s), io=12.0GiB (12.9GB), run=16429-16429msec
Run status group 4 (all jobs):
READ: bw=51.8MiB/s (54.3MB/s), 51.8MiB/s-51.8MiB/s (54.3MB/s-54.3MB/s), io=3107MiB (3258MB), run=60001-60001msec
Run status group 5 (all jobs):
WRITE: bw=285MiB/s (299MB/s), 285MiB/s-285MiB/s (299MB/s-299MB/s), io=12.0GiB (12.9GB), run=43101-43101msec
Disk stats (read/write):
dm-1: ios=3962397/6300960, merge=0/0, ticks=414994/80480, in_queue=495474, util=82.21%, aggrios=4044271/6391076, aggrmerge=4827/111, aggrticks=743085/180832, aggrin_queue=923917, aggrutil=79.16%
nvme1n1: ios=4044271/6391076, merge=4827/111, ticks=743085/180832, in_queue=923917, util=79.16%
Raw
ryzen-pve HC520 12T 7200rpm
root@ryzen-pve ➜ ~ fio --filename=/mnt/Disk3/fio.img scripts/fio/hd.fio
Read SEQ1M Q8T1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Read RND4K Q32T1: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Read RND4K Q1T1: (g=2): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Write SEQ1M Q8T1: (g=3): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
Write RND4K Q32T1: (g=4): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
Write RND4K Q1T1: (g=5): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 6 processes
Read SEQ1M Q8T1: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [_(5),w(1)][54.9%][w=2988KiB/s][w=747 IOPS][eta 04m:48s]
Read SEQ1M Q8T1: (groupid=0, jobs=1): err= 0: pid=3492783: Thu Mar 13 16:22:03 2025
read: IOPS=222, BW=222MiB/s (233MB/s)(12.0GiB/55289msec)
slat (usec): min=13, max=445, avg=22.98, stdev= 8.36
clat (msec): min=2, max=2193, avg=35.97, stdev=45.56
lat (msec): min=2, max=2193, avg=35.99, stdev=45.56
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 21], 10.00th=[ 25], 20.00th=[ 26],
| 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 37], 60.00th=[ 38],
| 70.00th=[ 40], 80.00th=[ 41], 90.00th=[ 45], 95.00th=[ 48],
| 99.00th=[ 56], 99.50th=[ 63], 99.90th=[ 667], 99.95th=[ 1133],
| 99.99th=[ 1720]
bw ( KiB/s): min=201915, max=249856, per=99.99%, avg=227557.72, stdev=12001.94, samples=110
iops : min= 197, max= 244, avg=222.15, stdev=11.73, samples=110
lat (msec) : 4=0.06%, 10=0.09%, 20=2.92%, 50=95.21%, 100=1.53%
lat (msec) : 750=0.10%, 2000=0.09%, >=2000=0.01%
cpu : usr=0.08%, sys=0.60%, ctx=12269, majf=4, minf=2060
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12288,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Read RND4K Q32T1: (groupid=1, jobs=1): err= 0: pid=3493218: Thu Mar 13 16:22:03 2025
read: IOPS=675, BW=2702KiB/s (2767kB/s)(159MiB/60102msec)
slat (nsec): min=2030, max=27920, avg=4798.05, stdev=2212.45
clat (usec): min=224, max=675797, avg=47368.36, stdev=55539.39
lat (usec): min=227, max=675800, avg=47373.16, stdev=55539.39
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 11],
| 30.00th=[ 15], 40.00th=[ 20], 50.00th=[ 28], 60.00th=[ 37],
| 70.00th=[ 51], 80.00th=[ 73], 90.00th=[ 115], 95.00th=[ 161],
| 99.00th=[ 268], 99.50th=[ 313], 99.90th=[ 422], 99.95th=[ 477],
| 99.99th=[ 542]
bw ( KiB/s): min= 1421, max= 2968, per=100.00%, avg=2704.28, stdev=220.95, samples=120
iops : min= 355, max= 742, avg=676.03, stdev=55.25, samples=120
lat (usec) : 250=0.01%, 500=0.17%, 750=0.01%
lat (msec) : 2=0.05%, 4=1.69%, 10=17.91%, 20=20.68%, 50=28.98%
lat (msec) : 100=17.72%, 250=11.51%, 500=1.24%, 750=0.04%
cpu : usr=0.15%, sys=0.46%, ctx=40564, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=40595,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Read RND4K Q1T1: (groupid=2, jobs=1): err= 0: pid=3493700: Thu Mar 13 16:22:03 2025
read: IOPS=180, BW=722KiB/s (739kB/s)(42.3MiB/60009msec)
slat (nsec): min=2130, max=33751, avg=7462.29, stdev=2516.07
clat (usec): min=194, max=35190, avg=5531.47, stdev=2503.07
lat (usec): min=202, max=35199, avg=5538.93, stdev=2503.08
clat percentiles (usec):
| 1.00th=[ 1336], 5.00th=[ 1778], 10.00th=[ 2180], 20.00th=[ 3032],
| 30.00th=[ 3851], 40.00th=[ 4686], 50.00th=[ 5473], 60.00th=[ 6325],
| 70.00th=[ 7177], 80.00th=[ 8029], 90.00th=[ 8848], 95.00th=[ 9241],
| 99.00th=[ 9634], 99.50th=[ 9765], 99.90th=[16909], 99.95th=[24773],
| 99.99th=[34866]
bw ( KiB/s): min= 648, max= 808, per=99.87%, avg=721.95, stdev=34.19, samples=119
iops : min= 162, max= 202, avg=180.46, stdev= 8.54, samples=119
lat (usec) : 250=0.13%, 750=0.01%, 1000=0.06%
lat (msec) : 2=7.52%, 4=23.94%, 10=68.22%, 20=0.04%, 50=0.07%
cpu : usr=0.09%, sys=0.16%, ctx=10832, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=10831,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Write SEQ1M Q8T1: (groupid=3, jobs=1): err= 0: pid=3494337: Thu Mar 13 16:22:03 2025
write: IOPS=231, BW=232MiB/s (243MB/s)(12.0GiB/53075msec); 0 zone resets
slat (usec): min=19, max=50559, avg=66.30, stdev=951.24
clat (msec): min=7, max=696, avg=34.48, stdev= 8.82
lat (msec): min=9, max=710, avg=34.55, stdev= 8.95
clat percentiles (msec):
| 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32],
| 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 37], 90.00th=[ 37], 95.00th=[ 42],
| 99.00th=[ 63], 99.50th=[ 74], 99.90th=[ 113], 99.95th=[ 124],
| 99.99th=[ 144]
bw ( KiB/s): min=186368, max=258565, per=99.97%, avg=237011.10, stdev=15203.10, samples=105
iops : min= 182, max= 252, avg=231.38, stdev=14.84, samples=105
lat (msec) : 10=0.02%, 20=0.18%, 50=97.40%, 100=2.16%, 250=0.23%
lat (msec) : 750=0.01%
cpu : usr=0.51%, sys=0.63%, ctx=12221, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.8%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,12288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Write RND4K Q32T1: (groupid=4, jobs=1): err= 0: pid=3494863: Thu Mar 13 16:22:03 2025
write: IOPS=822, BW=3289KiB/s (3368kB/s)(193MiB/60036msec); 0 zone resets
slat (nsec): min=1430, max=46157k, avg=13042.15, stdev=550750.86
clat (usec): min=488, max=416859, avg=38904.37, stdev=25264.33
lat (usec): min=840, max=416861, avg=38917.41, stdev=25259.07
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 13], 20.00th=[ 33],
| 30.00th=[ 36], 40.00th=[ 38], 50.00th=[ 40], 60.00th=[ 41],
| 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 50], 95.00th=[ 55],
| 99.00th=[ 79], 99.50th=[ 247], 99.90th=[ 342], 99.95th=[ 388],
| 99.99th=[ 405]
bw ( KiB/s): min= 1600, max=20400, per=99.00%, avg=3256.84, stdev=1889.79, samples=119
iops : min= 400, max= 5100, avg=814.15, stdev=472.44, samples=119
lat (usec) : 500=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.16%, 10=8.90%, 20=2.54%, 50=79.24%
lat (msec) : 100=8.35%, 250=0.30%, 500=0.48%
cpu : usr=0.13%, sys=0.47%, ctx=31789, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=0,49363,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Write RND4K Q1T1: (groupid=5, jobs=1): err= 0: pid=3495340: Thu Mar 13 16:22:03 2025
write: IOPS=741, BW=2967KiB/s (3038kB/s)(174MiB/60001msec); 0 zone resets
slat (nsec): min=2010, max=54911, avg=5319.01, stdev=2717.31
clat (usec): min=208, max=284291, avg=1341.93, stdev=4268.82
lat (usec): min=214, max=284295, avg=1347.25, stdev=4268.83
clat percentiles (usec):
| 1.00th=[ 293], 5.00th=[ 306], 10.00th=[ 310], 20.00th=[ 326],
| 30.00th=[ 865], 40.00th=[ 1057], 50.00th=[ 1123], 60.00th=[ 1270],
| 70.00th=[ 1434], 80.00th=[ 1696], 90.00th=[ 2180], 95.00th=[ 2704],
| 99.00th=[ 4555], 99.50th=[ 9634], 99.90th=[ 19792], 99.95th=[ 20317],
| 99.99th=[265290]
bw ( KiB/s): min= 1344, max= 3856, per=100.00%, avg=2968.05, stdev=483.15, samples=119
iops : min= 336, max= 964, avg=742.01, stdev=120.79, samples=119
lat (usec) : 250=0.16%, 500=25.81%, 750=3.04%, 1000=3.33%
lat (msec) : 2=54.79%, 4=11.49%, 10=0.92%, 20=0.39%, 50=0.04%
lat (msec) : 250=0.01%, 500=0.01%
cpu : usr=0.14%, sys=0.52%, ctx=44519, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,44507,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=12.0GiB (12.9GB), run=55289-55289msec
Run status group 1 (all jobs):
READ: bw=2702KiB/s (2767kB/s), 2702KiB/s-2702KiB/s (2767kB/s-2767kB/s), io=159MiB (166MB), run=60102-60102msec
Run status group 2 (all jobs):
READ: bw=722KiB/s (739kB/s), 722KiB/s-722KiB/s (739kB/s-739kB/s), io=42.3MiB (44.4MB), run=60009-60009msec
Run status group 3 (all jobs):
WRITE: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=12.0GiB (12.9GB), run=53075-53075msec
Run status group 4 (all jobs):
WRITE: bw=3289KiB/s (3368kB/s), 3289KiB/s-3289KiB/s (3368kB/s-3368kB/s), io=193MiB (202MB), run=60036-60036msec
Run status group 5 (all jobs):
WRITE: bw=2967KiB/s (3038kB/s), 2967KiB/s-2967KiB/s (3038kB/s-3038kB/s), io=174MiB (182MB), run=60001-60001msec
Disk stats (read/write):
sda: ios=76002/118343, merge=0/34, ticks=2835910/2834600, in_queue=5678521, util=98.81%
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment