Projekte

User Tools

Site Tools

Trace: • wireguard_access_to_home_network_using_pikvm
nas:performance:measurements_vm-hdd-disks

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nas:performance:measurements_vm-hdd-disks [2025/03/31 22:46] – adminnas:performance:measurements_vm-hdd-disks [2025/04/01 00:07] (current) – admin
Line 6: Line 6:
  
 ``` ```
 +zfs-2.2.7-pve2
 +zfs-kmod-2.2.7-pve2
 +
   pool: VM-HDD-Disks   pool: VM-HDD-Disks
  state: ONLINE  state: ONLINE
Line 60: Line 63:
 VM-HDD-Disks/subvol-113-disk-1  refquota              200G                             local VM-HDD-Disks/subvol-113-disk-1  refquota              200G                             local
 VM-HDD-Disks/subvol-113-disk-1  refreservation        none                             default VM-HDD-Disks/subvol-113-disk-1  refreservation        none                             default
-VM-HDD-Disks/subvol-113-disk-1  guid                  7307345886171180368              - 
 VM-HDD-Disks/subvol-113-disk-1  primarycache          all                              default VM-HDD-Disks/subvol-113-disk-1  primarycache          all                              default
 VM-HDD-Disks/subvol-113-disk-1  secondarycache        all                              default VM-HDD-Disks/subvol-113-disk-1  secondarycache        all                              default
-VM-HDD-Disks/subvol-113-disk-1  usedbysnapshots       0B                               - 
 VM-HDD-Disks/subvol-113-disk-1  usedbydataset         89.3G                            - VM-HDD-Disks/subvol-113-disk-1  usedbydataset         89.3G                            -
-VM-HDD-Disks/subvol-113-disk-1  usedbychildren        0B                               - 
-VM-HDD-Disks/subvol-113-disk-1  usedbyrefreservation  0B                               - 
 VM-HDD-Disks/subvol-113-disk-1  logbias               latency                          default VM-HDD-Disks/subvol-113-disk-1  logbias               latency                          default
 VM-HDD-Disks/subvol-113-disk-1  objsetid              67133                            - VM-HDD-Disks/subvol-113-disk-1  objsetid              67133                            -
Line 84: Line 83:
 VM-HDD-Disks/subvol-113-disk-1  snapdev               hidden                           default VM-HDD-Disks/subvol-113-disk-1  snapdev               hidden                           default
 VM-HDD-Disks/subvol-113-disk-1  acltype               posix                            local VM-HDD-Disks/subvol-113-disk-1  acltype               posix                            local
-VM-HDD-Disks/subvol-113-disk-1  context               none                             default 
-VM-HDD-Disks/subvol-113-disk-1  fscontext             none                             default 
-VM-HDD-Disks/subvol-113-disk-1  defcontext            none                             default 
-VM-HDD-Disks/subvol-113-disk-1  rootcontext           none                             default 
 VM-HDD-Disks/subvol-113-disk-1  relatime              on                               default VM-HDD-Disks/subvol-113-disk-1  relatime              on                               default
 VM-HDD-Disks/subvol-113-disk-1  redundant_metadata    all                              default VM-HDD-Disks/subvol-113-disk-1  redundant_metadata    all                              default
 VM-HDD-Disks/subvol-113-disk-1  overlay               on                               default VM-HDD-Disks/subvol-113-disk-1  overlay               on                               default
 VM-HDD-Disks/subvol-113-disk-1  encryption            off                              default VM-HDD-Disks/subvol-113-disk-1  encryption            off                              default
-VM-HDD-Disks/subvol-113-disk-1  keylocation           none                             default 
-VM-HDD-Disks/subvol-113-disk-1  keyformat             none                             default 
-VM-HDD-Disks/subvol-113-disk-1  pbkdf2iters           0                                default 
 VM-HDD-Disks/subvol-113-disk-1  special_small_blocks  64K                              local VM-HDD-Disks/subvol-113-disk-1  special_small_blocks  64K                              local
 VM-HDD-Disks/subvol-113-disk-1  prefetch              all                              default VM-HDD-Disks/subvol-113-disk-1  prefetch              all                              default
Line 101: Line 93:
 ## Direct ## Direct
  
 +* large file, small writes: ca. 250 IOPS (Disks) `iops        : min=   34, max=  358, avg=256.90, stdev=16.70, samples=477`
 +* small file, small writes: ca. 413k IOPS (RAM, async writes) `iops        : min=391884, max=421246, avg=413054.47, stdev=1192.75, samples=476`
 +* small file, small writes, --fsync=1: ca. 3400 IOPS (NVMe/SSD, sync writes) `iops        : min= 1910, max= 4776, avg=3462.39, stdev=191.88, samples=476`
 +
 +NFS usually uses sync writes.
 +
 +```
 +IOPS-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [w(4)][100.0%][w=996KiB/s][w=249 IOPS][eta 00m:00s]
 +IOPS-write: (groupid=0, jobs=4): err= 0: pid=53311: Mon Mar 31 22:53:08 2025
 +  write: IOPS=256, BW=1028KiB/s (1052kB/s)(60.4MiB/60132msec); 0 zone resets
 +    slat (usec): min=7, max=430036, avg=15553.41, stdev=20371.38
 +    clat (usec): min=6, max=1070.4k, avg=480135.61, stdev=138489.39
 +     lat (msec): min=215, max=1070, avg=495.69, stdev=140.38
 +    clat percentiles (msec):
 +     |  1.00th=[  317],  5.00th=[  351], 10.00th=[  368], 20.00th=[  393],
 +     | 30.00th=[  414], 40.00th=[  430], 50.00th=[  447], 60.00th=[  464],
 +     | 70.00th=[  485], 80.00th=[  510], 90.00th=[  600], 95.00th=[  869],
 +     | 99.00th=[  961], 99.50th=[  978], 99.90th=[ 1011], 99.95th=[ 1028],
 +     | 99.99th=[ 1053]
 +   bw (  KiB/s): min=  136, max= 1432, per=99.93%, avg=1027.62, stdev=66.80, samples=477
 +   iops        : min=   34, max=  358, avg=256.90, stdev=16.70, samples=477
 +  lat (usec)   : 10=0.03%
 +  lat (msec)   : 250=0.02%, 500=76.67%, 750=14.71%, 1000=8.41%, 2000=0.16%
 +  cpu          : usr=0.05%, sys=0.45%, ctx=14272, majf=0, minf=47
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,15450,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=1028KiB/s (1052kB/s), 1028KiB/s-1028KiB/s (1052kB/s-1052kB/s), io=60.4MiB (63.3MB), run=60132-60132msec
 +
 +IOPS-write-s: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [w(4)][100.0%][w=1623MiB/s][w=416k IOPS][eta 00m:00s]
 +IOPS-write-s: (groupid=0, jobs=4): err= 0: pid=53358: Mon Mar 31 22:54:08 2025
 +  write: IOPS=413k, BW=1613MiB/s (1691MB/s)(94.5GiB/60002msec); 0 zone resets
 +    slat (usec): min=6, max=27440, avg= 7.84, stdev= 8.13
 +    clat (nsec): min=1483, max=3674.3k, avg=300827.73, stdev=16636.22
 +     lat (usec): min=8, max=27764, avg=308.67, stdev=18.78
 +    clat percentiles (usec):
 +     |  1.00th=[  285],  5.00th=[  289], 10.00th=[  289], 20.00th=[  293],
 +     | 30.00th=[  293], 40.00th=[  293], 50.00th=[  297], 60.00th=[  297],
 +     | 70.00th=[  302], 80.00th=[  306], 90.00th=[  314], 95.00th=[  326],
 +     | 99.00th=[  371], 99.50th=[  383], 99.90th=[  420], 99.95th=[  445],
 +     | 99.99th=[  529]
 +   bw (  MiB/s): min= 1530, max= 1645, per=100.00%, avg=1613.49, stdev= 4.66, samples=476
 +   iops        : min=391884, max=421246, avg=413054.47, stdev=1192.75, samples=476
 +  lat (usec)   : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
 +  lat (usec)   : 500=99.98%, 750=0.01%, 1000=0.01%
 +  lat (msec)   : 2=0.01%, 4=0.01%
 +  cpu          : usr=16.85%, sys=83.08%, ctx=2413, majf=0, minf=45
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,24778203,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=1613MiB/s (1691MB/s), 1613MiB/s-1613MiB/s (1691MB/s-1691MB/s), io=94.5GiB (101GB), run=60002-60002msec
 +IOPS-write-s: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [w(4)][100.0%][w=13.9MiB/s][w=3571 IOPS][eta 00m:00s]
 +IOPS-write-s: (groupid=0, jobs=4): err= 0: pid=53642: Mon Mar 31 23:01:58 2025
 +  write: IOPS=3455, BW=13.5MiB/s (14.2MB/s)(810MiB/60001msec); 0 zone resets
 +    slat (usec): min=7, max=460, avg=12.45, stdev= 4.02
 +    clat (usec): min=951, max=206164, avg=35885.64, stdev=22487.02
 +     lat (usec): min=968, max=206179, avg=35898.09, stdev=22487.24
 +    clat percentiles (msec):
 +     |  1.00th=[   26],  5.00th=[   26], 10.00th=[   27], 20.00th=[   27],
 +     | 30.00th=[   27], 40.00th=[   27], 50.00th=[   28], 60.00th=[   29],
 +     | 70.00th=[   29], 80.00th=[   38], 90.00th=[   53], 95.00th=[  103],
 +     | 99.00th=[  122], 99.50th=[  178], 99.90th=[  203], 99.95th=[  203],
 +     | 99.99th=[  205]
 +   bw (  KiB/s): min= 7640, max=19104, per=100.00%, avg=13849.55, stdev=767.50, samples=476
 +   iops        : min= 1910, max= 4776, avg=3462.39, stdev=191.88, samples=476
 +  lat (usec)   : 1000=0.01%
 +  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.03%, 50=86.37%
 +  lat (msec)   : 100=8.09%, 250=5.48%
 +  fsync/fdatasync/sync_file_range:
 +    sync (msec): min=3, max=93767k, avg=1847.24, stdev=411987.63
 +    sync percentiles (msec):
 +     |  1.00th=[   27],  5.00th=[   27], 10.00th=[   27], 20.00th=[   28],
 +     | 30.00th=[   28], 40.00th=[   28], 50.00th=[   29], 60.00th=[   30],
 +     | 70.00th=[   30], 80.00th=[   40], 90.00th=[   55], 95.00th=[  104],
 +     | 99.00th=[  123], 99.50th=[  182], 99.90th=[  205], 99.95th=[  205],
 +     | 99.99th=[  207]
 +  cpu          : usr=0.43%, sys=3.03%, ctx=414534, majf=0, minf=49
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=199.9%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,207316,0,207196 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=13.5MiB/s (14.2MB/s), 13.5MiB/s-13.5MiB/s (14.2MB/s-14.2MB/s), io=810MiB (849MB), run=60001-60001msec
 +```
 +* large file, large writes: ca. 1890 MiB/s (Disks, starves) `bw (  MiB/s): min=    9, max= 4147, per=100.00%, avg=1890.35, stdev=361.83, samples=104`
 +* large file, large writes, --fsync=1: ca. 215 MiB/s (Disks, high stddev) `bw (  KiB/s): min=40980, max=829440, per=99.06%, avg=216350.45, stdev=34031.22, samples=476`
 +* small file, large writes: ca. 8400 MiB/s (NVMe/SSD) `bw (  MiB/s): min= 2092, max= 9335, per=99.06%, avg=8472.71, stdev=201.91, samples=476`
 +
 +```
 +IOPS-write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [W(4)][100.0%][w=1864MiB/s][w=1863 IOPS][eta 00m:00s]
 +IOPS-write: (groupid=0, jobs=4): err= 0: pid=54143: Mon Mar 31 23:15:08 2025
 +  write: IOPS=406, BW=406MiB/s (426MB/s)(24.1GiB/60877msec); 0 zone resets
 +    slat (usec): min=115, max=8273.0k, avg=9440.09, stdev=235906.05
 +    clat (usec): min=9, max=8667.4k, avg=304834.10, stdev=1303425.91
 +     lat (msec): min=16, max=8668, avg=314.27, stdev=1323.50
 +    clat percentiles (msec):
 +     |  1.00th=[   20],  5.00th=[   22], 10.00th=[   23], 20.00th=[   25],
 +     | 30.00th=[   27], 40.00th=[   29], 50.00th=[   31], 60.00th=[   34],
 +     | 70.00th=[   36], 80.00th=[   40], 90.00th=[   53], 95.00th=[ 1435],
 +     | 99.00th=[ 7819], 99.50th=[ 8288], 99.90th=[ 8658], 99.95th=[ 8658],
 +     | 99.99th=[ 8658]
 +   bw (  MiB/s): min=    9, max= 4147, per=100.00%, avg=1890.35, stdev=361.83, samples=104
 +   iops        : min=    6, max= 4147, avg=1889.65, stdev=361.89, samples=104
 +  lat (usec)   : 10=0.01%, 20=0.01%
 +  lat (msec)   : 20=2.71%, 50=86.75%, 100=3.41%, 250=1.08%, 500=0.02%
 +  lat (msec)   : 750=0.01%, 1000=0.49%, 2000=2.01%, >=2000=3.51%
 +  cpu          : usr=3.69%, sys=4.19%, ctx=40593, majf=0, minf=46
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,24728,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=406MiB/s (426MB/s), 406MiB/s-406MiB/s (426MB/s-426MB/s), io=24.1GiB (25.9GB), run=60877-60877msec
 +IOPS-write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [W(4)][100.0%][w=190MiB/s][w=190 IOPS][eta 00m:00s]
 +IOPS-write: (groupid=0, jobs=4): err= 0: pid=54259: Mon Mar 31 23:18:07 2025
 +  write: IOPS=213, BW=213MiB/s (224MB/s)(12.5GiB/60014msec); 0 zone resets
 +    slat (usec): min=104, max=35746, avg=265.87, stdev=626.42
 +    clat (msec): min=13, max=2716, avg=580.48, stdev=324.43
 +     lat (msec): min=13, max=2716, avg=580.74, stdev=324.41
 +    clat percentiles (msec):
 +     |  1.00th=[  138],  5.00th=[  144], 10.00th=[  159], 20.00th=[  232],
 +     | 30.00th=[  567], 40.00th=[  600], 50.00th=[  625], 60.00th=[  651],
 +     | 70.00th=[  684], 80.00th=[  709], 90.00th=[  726], 95.00th=[  760],
 +     | 99.00th=[ 2467], 99.50th=[ 2567], 99.90th=[ 2635], 99.95th=[ 2635],
 +     | 99.99th=[ 2702]
 +   bw (  KiB/s): min=40980, max=829440, per=99.06%, avg=216350.45, stdev=34031.22, samples=476
 +   iops        : min=   40, max=  810, avg=211.26, stdev=33.23, samples=476
 +  lat (msec)   : 20=0.03%, 50=0.08%, 100=0.35%, 250=20.99%, 500=2.00%
 +  lat (msec)   : 750=70.84%, 1000=3.23%, 2000=1.03%, >=2000=1.45%
 +  fsync/fdatasync/sync_file_range:
 +    sync (msec): min=48, max=94736k, avg=30486.37, stdev=1682403.65
 +    sync percentiles (msec):
 +     |  1.00th=[  142],  5.00th=[  150], 10.00th=[  165], 20.00th=[  239],
 +     | 30.00th=[  592], 40.00th=[  617], 50.00th=[  651], 60.00th=[  676],
 +     | 70.00th=[  709], 80.00th=[  726], 90.00th=[  743], 95.00th=[  776],
 +     | 99.00th=[ 2567], 99.50th=[ 2635], 99.90th=[ 2735], 99.95th=[ 2802],
 +     | 99.99th=[17113]
 +  cpu          : usr=1.41%, sys=2.79%, ctx=30119, majf=0, minf=46
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.5%, 32=198.1%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,12800,0,12680 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=213MiB/s (224MB/s), 213MiB/s-213MiB/s (224MB/s-224MB/s), io=12.5GiB (13.4GB), run=60014-60014msec
 +IOPS-write-s: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
 +...
 +fio-3.33
 +Starting 4 processes
 +Jobs: 4 (f=4): [W(4)][100.0%][w=8896MiB/s][w=71.2k IOPS][eta 00m:00s]
 +IOPS-write-s: (groupid=0, jobs=4): err= 0: pid=53916: Mon Mar 31 23:08:53 2025
 +  write: IOPS=68.4k, BW=8553MiB/s (8968MB/s)(496GiB/59435msec); 0 zone resets
 +    slat (usec): min=10, max=4097, avg=13.98, stdev=17.68
 +    clat (nsec): min=831, max=183134, avg=1011.48, stdev=405.83
 +     lat (usec): min=11, max=4107, avg=14.99, stdev=17.74
 +    clat percentiles (nsec):
 +     |  1.00th=[  868],  5.00th=[  884], 10.00th=[  892], 20.00th=[  900],
 +     | 30.00th=[  916], 40.00th=[  924], 50.00th=[  932], 60.00th=[  948],
 +     | 70.00th=[  972], 80.00th=[ 1004], 90.00th=[ 1160], 95.00th=[ 1368],
 +     | 99.00th=[ 2448], 99.50th=[ 2736], 99.90th=[ 5472], 99.95th=[ 7264],
 +     | 99.99th=[10560]
 +   bw (  MiB/s): min= 2092, max= 9335, per=99.06%, avg=8472.71, stdev=201.91, samples=476
 +   iops        : min=16742, max=74682, avg=67781.66, stdev=1615.31, samples=476
 +  lat (nsec)   : 1000=78.37%
 +  lat (usec)   : 2=19.94%, 4=1.55%, 10=0.12%, 20=0.01%, 50=0.01%
 +  lat (usec)   : 100=0.01%, 250=0.01%
 +  cpu          : usr=52.18%, sys=38.05%, ctx=47779, majf=0, minf=45
 +  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,4066760,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=8553MiB/s (8968MB/s), 8553MiB/s-8553MiB/s (8968MB/s-8968MB/s), io=496GiB (533GB), run=59435-59435msec
 +```
 +* large file, small writes: 7 ms - 230 ms (Disks) `lat (usec): min=9, max=233541, avg=7005.35, stdev=8208.20`
 +* small file, small writes: ca. 7 us `lat (usec): min=7, max=236, avg= 7.85, stdev= 1.05`
 +```
 +IOPS-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
 +fio-3.33
 +Starting 1 process
 +Jobs: 1 (f=1): [w(1)][100.0%][w=548KiB/s][w=137 IOPS][eta 00m:00s]
 +IOPS-write: (groupid=0, jobs=1): err= 0: pid=54699: Mon Mar 31 23:30:35 2025
 +  write: IOPS=142, BW=571KiB/s (584kB/s)(33.4MiB/60006msec); 0 zone resets
 +    slat (usec): min=9, max=233535, avg=7003.18, stdev=8207.58
 +    clat (nsec): min=641, max=49263, avg=2172.48, stdev=2057.69
 +     lat (usec): min=9, max=233541, avg=7005.35, stdev=8208.20
 +    clat percentiles (nsec):
 +     |  1.00th=[  700],  5.00th=[  780], 10.00th=[  828], 20.00th=[ 1208],
 +     | 30.00th=[ 1640], 40.00th=[ 1784], 50.00th=[ 1832], 60.00th=[ 1880],
 +     | 70.00th=[ 2008], 80.00th=[ 2320], 90.00th=[ 3440], 95.00th=[ 5088],
 +     | 99.00th=[ 8896], 99.50th=[14528], 99.90th=[25984], 99.95th=[33536],
 +     | 99.99th=[49408]
 +   bw (  KiB/s): min=  192, max=  848, per=99.87%, avg=570.76, stdev=132.77, samples=119
 +   iops        : min=   48, max=  212, avg=142.69, stdev=33.19, samples=119
 +  lat (nsec)   : 750=2.58%, 1000=10.41%
 +  lat (usec)   : 2=56.51%, 4=22.76%, 10=6.87%, 20=0.62%, 50=0.26%
 +  cpu          : usr=0.09%, sys=0.93%, ctx=7523, majf=0, minf=9
 +  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,8562,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=1
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=571KiB/s (584kB/s), 571KiB/s-571KiB/s (584kB/s-584kB/s), io=33.4MiB (35.1MB), run=60006-60006msec
 +IOPS-write-s: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
 +fio-3.33
 +Starting 1 process
 +Jobs: 1 (f=1): [w(1)][100.0%][w=428MiB/s][w=110k IOPS][eta 00m:00s]
 +IOPS-write-s: (groupid=0, jobs=1): err= 0: pid=54745: Mon Mar 31 23:31:35 2025
 +  write: IOPS=111k, BW=434MiB/s (455MB/s)(25.4GiB/60001msec); 0 zone resets
 +    slat (usec): min=6, max=235, avg= 7.19, stdev= 1.00
 +    clat (nsec): min=611, max=57498, avg=653.88, stdev=218.22
 +     lat (usec): min=7, max=236, avg= 7.85, stdev= 1.05
 +    clat percentiles (nsec):
 +     |  1.00th=[  620],  5.00th=[  628], 10.00th=[  628], 20.00th=[  628],
 +     | 30.00th=[  628], 40.00th=[  644], 50.00th=[  644], 60.00th=[  644],
 +     | 70.00th=[  644], 80.00th=[  644], 90.00th=[  652], 95.00th=[  732],
 +     | 99.00th=[  852], 99.50th=[  932], 99.90th=[ 1832], 99.95th=[ 5792],
 +     | 99.99th=[ 9408]
 +   bw (  KiB/s): min=429576, max=452824, per=100.00%, avg=444288.81, stdev=4681.27, samples=119
 +   iops        : min=107394, max=113206, avg=111072.22, stdev=1170.28, samples=119
 +  lat (nsec)   : 750=95.66%, 1000=3.99%
 +  lat (usec)   : 2=0.27%, 4=0.01%, 10=0.06%, 20=0.01%, 50=0.01%
 +  lat (usec)   : 100=0.01%
 +  cpu          : usr=17.49%, sys=82.50%, ctx=481, majf=0, minf=11
 +  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,6662036,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=1
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=434MiB/s (455MB/s), 434MiB/s-434MiB/s (455MB/s-455MB/s), io=25.4GiB (27.3GB), run=60001-60001msec
 +```
 ## NFS 10GB ## NFS 10GB
 +
 +* large file, small writes: `iops        : min=   16, max=  384, avg=301.21, stdev=20.70, samples=478`
 +* small file, small writed: `iops        : min= 3044, max= 5952, avg=4664.82, stdev=154.45, samples=480`
 +
 +```
 +IOPS-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
 +...
 +fio-3.28
 +Starting 4 processes
 +Jobs: 4 (f=4): [w(4)][100.0%][w=1416KiB/s][w=354 IOPS][eta 00m:00s]
 +IOPS-write: (groupid=0, jobs=4): err= 0: pid=1521489: Tue Apr  1 00:01:54 2025
 +  write: IOPS=300, BW=1201KiB/s (1230kB/s)(70.8MiB/60350msec); 0 zone resets
 +    slat (usec): min=2, max=16486, avg=10.54, stdev=172.69
 +    clat (msec): min=29, max=1058, avg=426.06, stdev=148.93
 +     lat (msec): min=29, max=1058, avg=426.07, stdev=148.92
 +    clat percentiles (msec):
 +     |  1.00th=[  317],  5.00th=[  334], 10.00th=[  342], 20.00th=[  355],
 +     | 30.00th=[  363], 40.00th=[  372], 50.00th=[  380], 60.00th=[  388],
 +     | 70.00th=[  401], 80.00th=[  430], 90.00th=[  514], 95.00th=[  894],
 +     | 99.00th=[  978], 99.50th=[  995], 99.90th=[ 1028], 99.95th=[ 1036],
 +     | 99.99th=[ 1045]
 +   bw (  KiB/s): min=   64, max= 1536, per=100.00%, avg=1204.94, stdev=82.81, samples=478
 +   iops        : min=   16, max=  384, avg=301.21, stdev=20.70, samples=478
 +  lat (msec)   : 50=0.06%, 100=0.06%, 250=0.18%, 500=88.50%, 750=3.56%
 +  lat (msec)   : 1000=7.19%, 2000=0.46%
 +  cpu          : usr=0.03%, sys=0.09%, ctx=18310, majf=0, minf=44
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.3%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,18124,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=1201KiB/s (1230kB/s), 1201KiB/s-1201KiB/s (1230kB/s-1230kB/s), io=70.8MiB (74.2MB), run=60350-60350msec
 +IOPS-write-s: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
 +...
 +fio-3.28
 +Starting 4 processes
 +Jobs: 4 (f=4): [w(4)][100.0%][w=17.2MiB/s][w=4394 IOPS][eta 00m:00s]
 +IOPS-write-s: (groupid=0, jobs=4): err= 0: pid=1523614: Tue Apr  1 00:04:34 2025
 +  write: IOPS=4658, BW=18.2MiB/s (19.1MB/s)(1094MiB/60104msec); 0 zone resets
 +    slat (nsec): min=1693, max=583438, avg=3840.63, stdev=2322.46
 +    clat (usec): min=1146, max=180128, avg=27467.80, stdev=17714.52
 +     lat (usec): min=1157, max=180132, avg=27471.73, stdev=17714.55
 +    clat percentiles (msec):
 +     |  1.00th=[   18],  5.00th=[   21], 10.00th=[   22], 20.00th=[   22],
 +     | 30.00th=[   23], 40.00th=[   23], 50.00th=[   24], 60.00th=[   24],
 +     | 70.00th=[   25], 80.00th=[   26], 90.00th=[   27], 95.00th=[   92],
 +     | 99.00th=[   99], 99.50th=[  105], 99.90th=[  176], 99.95th=[  178],
 +     | 99.99th=[  180]
 +   bw (  KiB/s): min=12176, max=23808, per=100.00%, avg=18659.27, stdev=617.79, samples=480
 +   iops        : min= 3044, max= 5952, avg=4664.82, stdev=154.45, samples=480
 +  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=4.00%, 50=90.84%
 +  lat (msec)   : 100=4.50%, 250=0.65%
 +  cpu          : usr=0.32%, sys=0.74%, ctx=272757, majf=0, minf=52
 +  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
 +     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 +     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
 +     issued rwts: total=0,280013,0,0 short=0,0,0,0 dropped=0,0,0,0
 +     latency   : target=0, window=0, percentile=100.00%, depth=32
 +
 +Run status group 0 (all jobs):
 +  WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=1094MiB (1147MB), run=60104-60104msec
 +```
nas/performance/measurements_vm-hdd-disks.1743453970.txt.gz · Last modified: by admin