# Measurements vmhost storage ``` zfs-2.3.1-1~bpo12+1 zfs-kmod-2.3.1-1~bpo12+1 pool: storage state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 1 days 17:34:53 with 0 errors on Mon Mar 10 17:58:55 2025 config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT6WGYM ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT2SJZL ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT6K3PP ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT6XMPM ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT2SYB0 ONLINE 0 0 0 ata-ST20000NM007D-3DJ103_ZVT7F7KT ONLINE 0 0 0 NAME PROPERTY VALUE SOURCE storage/gog-offline type filesystem - storage/gog-offline creation Fr Feb 7 13:14 2025 - storage/gog-offline used 316G - storage/gog-offline available 14.0T - storage/gog-offline referenced 316G - storage/gog-offline compressratio 1.00x - storage/gog-offline mounted yes - storage/gog-offline quota none default storage/gog-offline reservation none default storage/gog-offline recordsize 4K inherited from storage storage/gog-offline mountpoint /storage/gog-offline default storage/gog-offline sharenfs off default storage/gog-offline checksum on default storage/gog-offline compression on default storage/gog-offline atime on default storage/gog-offline devices on default storage/gog-offline exec on default storage/gog-offline setuid on default storage/gog-offline readonly off default storage/gog-offline zoned off default storage/gog-offline snapdir hidden default storage/gog-offline aclmode discard default storage/gog-offline aclinherit restricted default storage/gog-offline createtxg 27126575 - storage/gog-offline canmount on default storage/gog-offline xattr on default storage/gog-offline copies 1 default storage/gog-offline version 5 - storage/gog-offline utf8only off - storage/gog-offline normalization none - storage/gog-offline casesensitivity sensitive - storage/gog-offline vscan off default storage/gog-offline nbmand off default storage/gog-offline sharesmb off default storage/gog-offline refquota none default storage/gog-offline refreservation none default storage/gog-offline guid 3191482249681275415 - storage/gog-offline primarycache all default storage/gog-offline secondarycache all default storage/gog-offline usedbysnapshots 780K - storage/gog-offline usedbydataset 316G - storage/gog-offline usedbychildren 0B - storage/gog-offline usedbyrefreservation 0B - storage/gog-offline logbias latency default storage/gog-offline objsetid 148271 - storage/gog-offline dedup off default storage/gog-offline mlslabel none default storage/gog-offline sync standard default storage/gog-offline dnodesize legacy default storage/gog-offline refcompressratio 1.00x - storage/gog-offline written 65.0G - storage/gog-offline logicalused 292G - storage/gog-offline logicalreferenced 292G - storage/gog-offline volmode default default storage/gog-offline filesystem_limit none default storage/gog-offline snapshot_limit none default storage/gog-offline filesystem_count none default storage/gog-offline snapshot_count none default storage/gog-offline snapdev hidden default storage/gog-offline acltype off default storage/gog-offline context none default storage/gog-offline fscontext none default storage/gog-offline defcontext none default storage/gog-offline rootcontext none default storage/gog-offline relatime on default storage/gog-offline redundant_metadata all default storage/gog-offline overlay on default storage/gog-offline encryption aes-256-gcm - storage/gog-offline keylocation none default storage/gog-offline keyformat hex - storage/gog-offline pbkdf2iters 0 default storage/gog-offline encryptionroot storage - storage/gog-offline keystatus available - storage/gog-offline special_small_blocks 0 default storage/gog-offline snapshots_changed Fr Feb 7 13:26:25 2025 - storage/gog-offline prefetch all default storage/gog-offline direct standard default storage/gog-offline longname off default ``` ## Direct ``` IOPS-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... fio-3.28 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=36.0MiB/s][w=9212 IOPS][eta 00m:00s] IOPS-write: (groupid=0, jobs=4): err= 0: pid=1515224: Mon Mar 31 23:54:44 2025 write: IOPS=4225, BW=16.5MiB/s (17.3MB/s)(990MiB/60001msec); 0 zone resets slat (usec): min=122, max=740375, avg=943.71, stdev=10962.59 clat (usec): min=2, max=1035.9k, avg=29277.62, stdev=76886.07 lat (usec): min=1263, max=1072.4k, avg=30221.54, stdev=78319.38 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 13], 80.00th=[ 27], 90.00th=[ 59], 95.00th=[ 88], | 99.00th=[ 558], 99.50th=[ 667], 99.90th=[ 827], 99.95th=[ 911], | 99.99th=[ 995] bw ( KiB/s): min= 72, max=54168, per=100.00%, avg=16977.71, stdev=3725.64, samples=474 iops : min= 18, max=13542, avg=4244.41, stdev=931.40, samples=474 lat (usec) : 4=0.01%, 10=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=45.18%, 20=31.26%, 50=11.34% lat (msec) : 100=8.14%, 250=2.67%, 500=0.21%, 750=1.02%, 1000=0.17% lat (msec) : 2000=0.01% cpu : usr=0.32%, sys=6.34%, ctx=270565, majf=2, minf=59 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,253519,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=990MiB (1038MB), run=60001-60001msec IOPS-write-s: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... fio-3.28 Starting 4 processes Jobs: 4 (f=4): [w(4)][100.0%][w=58.2MiB/s][w=14.9k IOPS][eta 00m:00s] IOPS-write-s: (groupid=0, jobs=4): err= 0: pid=1515912: Mon Mar 31 23:55:44 2025 write: IOPS=11.8k, BW=46.2MiB/s (48.4MB/s)(2770MiB/60001msec); 0 zone resets slat (usec): min=62, max=333152, avg=336.07, stdev=1393.43 clat (nsec): min=1273, max=268361k, avg=10422898.83, stdev=11578471.24 lat (usec): min=774, max=414802, avg=10759.14, stdev=11874.60 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 18], | 99.00th=[ 64], 99.50th=[ 105], 99.90th=[ 153], 99.95th=[ 174], | 99.99th=[ 255] bw ( KiB/s): min= 4832, max=64656, per=99.81%, avg=47190.05, stdev=3745.76, samples=476 iops : min= 1208, max=16164, avg=11797.51, stdev=936.44, samples=476 lat (usec) : 2=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=77.89%, 20=18.53%, 50=2.29% lat (msec) : 100=0.70%, 250=0.56%, 500=0.02% cpu : usr=0.70%, sys=12.39%, ctx=724689, majf=0, minf=62 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,709176,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=46.2MiB/s (48.4MB/s), 46.2MiB/s-46.2MiB/s (48.4MB/s-48.4MB/s), io=2770MiB (2905MB), run=60001-60001msec ```