Write bandwidth
[kindr@nano8 projects.link]$ fio --name=write_bandwidth_test --filename=/mnt/projects/grp_202/kindr/test --filesize=2000G --time_based --ramp_time=2s --runtime=1m --ioengine=libaio --direct=0 --verify=0 --randrepeat=0 --bs=1M --iodepth=32 --rw=randwrite
write_bandwidth_test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.1
Starting 1 process
write_bandwidth_test: Laying out IO file (1 file / 2048000MiB)
fio: native_fallocate call failed: Operation not supported
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=490MiB/s][r=0,w=490 IOPS][eta 00m:00s]
write_bandwidth_test: (groupid=0, jobs=1): err= 0: pid=18556: Wed Jul 25 10:43:41 2018
write: IOPS=455, BW=456MiB/s (478MB/s)(26.7GiB/60003msec)
slat (usec): min=1094, max=236543, avg=2189.88, stdev=4388.83
clat (usec): min=8, max=425501, avg=68070.66, stdev=30008.93
lat (msec): min=3, max=427, avg=70.26, stdev=30.59
clat percentiles (msec):
| 1.00th=[ 47], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 55],
| 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63],
| 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 84], 95.00th=[ 131],
| 99.00th=[ 184], 99.50th=[ 222], 99.90th=[ 418], 99.95th=[ 422],
| 99.99th=[ 426]
bw ( KiB/s): min=153082, max=652955, per=100.00%, avg=470188.13, stdev=94818.54, samples=120
iops : min= 149, max= 637, avg=458.71, stdev=92.60, samples=120
lat (usec) : 10=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=3.53%, 100=90.16%
lat (msec) : 250=5.98%, 500=0.40%
cpu : usr=3.84%, sys=14.89%, ctx=27906, majf=0, minf=3038
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=103.4%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,27309,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s), io=26.7GiB (28.7GB), run=60003-60003msec
Write IOPS
[kindr@nano8 projects.link]$ fio --name=write_iops_test --filename=/mnt/projects/grp_202/kindr/test1 --filesize=2000G --time_based --ramp_time=2s --runtime=1m --ioengine=libaio --direct=0 --verify=0 --randrepeat=0 --bs=4K --iodepth=64 --rw=randwrite
write_iops_test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
write_iops_test: Laying out IO file (1 file / 2048000MiB)
fio: native_fallocate call failed: Operation not supported
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=9313KiB/s][r=0,w=2328 IOPS][eta 00m:00s]
write_iops_test: (groupid=0, jobs=1): err= 0: pid=25697: Wed Jul 25 10:50:35 2018
write: IOPS=2323, BW=9298KiB/s (9521kB/s)(545MiB/60001msec)
slat (usec): min=123, max=115243, avg=423.60, stdev=1272.53
clat (usec): min=7, max=171943, avg=27109.88, stdev=11071.71
lat (usec): min=1147, max=172299, avg=27534.34, stdev=11167.70
clat percentiles (msec):
| 1.00th=[ 20], 5.00th=[ 22], 10.00th=[ 23], 20.00th=[ 24],
| 30.00th=[ 25], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 27],
| 70.00th=[ 28], 80.00th=[ 28], 90.00th=[ 30], 95.00th=[ 33],
| 99.00th=[ 80], 99.50th=[ 127], 99.90th=[ 133], 99.95th=[ 146],
| 99.99th=[ 163]
bw ( KiB/s): min= 6188, max=12248, per=100.00%, avg=9368.40, stdev=1169.87, samples=120
iops : min= 1547, max= 3062, avg=2341.74, stdev=292.46, samples=120
lat (usec) : 10=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=2.04%, 50=96.12%
lat (msec) : 100=0.92%, 250=0.95%
cpu : usr=1.89%, sys=9.31%, ctx=139786, majf=0, minf=3454
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=0,139411,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=9298KiB/s (9521kB/s), 9298KiB/s-9298KiB/s (9521kB/s-9521kB/s), io=545MiB (571MB), run=60001-60001msec
Write latency
[kindr@nano8 projects.link]$ fio --name=write_latency_test --filename=/mnt/projects/grp_202/kindr/test2 --filesize=2000G --time_based --ramp_time=2s --runtime=1m --ioengine=libaio --direct=0 --verify=0 --randrepeat=0 --bs=4K --iodepth=4 --rw=randwrite
write_latency_test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.1
Starting 1 process
write_latency_test: Laying out IO file (1 file / 2048000MiB)
fio: native_fallocate call failed: Operation not supported
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=9376KiB/s][r=0,w=2344 IOPS][eta 00m:00s]
write_latency_test: (groupid=0, jobs=1): err= 0: pid=26266: Wed Jul 25 10:58:34 2018
write: IOPS=2319, BW=9276KiB/s (9499kB/s)(544MiB/60001msec)
slat (usec): min=127, max=109715, avg=424.57, stdev=1227.60
clat (usec): min=4, max=113436, avg=1295.70, stdev=2139.78
lat (usec): min=442, max=114090, avg=1721.16, stdev=2476.65
clat percentiles (usec):
| 1.00th=[ 775], 5.00th=[ 898], 10.00th=[ 971], 20.00th=[ 1057],
| 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1270],
| 70.00th=[ 1319], 80.00th=[ 1385], 90.00th=[ 1500], 95.00th=[ 1614],
| 99.00th=[ 2245], 99.50th=[ 2966], 99.90th=[ 10159], 99.95th=[ 21627],
| 99.99th=[102237]
bw ( KiB/s): min= 5629, max=11214, per=100.00%, avg=9346.92, stdev=1072.46, samples=119
iops : min= 1407, max= 2803, avg=2336.46, stdev=268.10, samples=119
lat (usec) : 10=0.01%, 500=0.01%, 750=0.70%, 1000=12.56%
lat (msec) : 2=85.33%, 4=1.09%, 10=0.21%, 20=0.05%, 50=0.01%
lat (msec) : 250=0.04%
cpu : usr=1.87%, sys=9.31%, ctx=139633, majf=0, minf=1604
IO depths : 1=0.1%, 2=0.1%, 4=103.3%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,139144,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: bw=9276KiB/s (9499kB/s), 9276KiB/s-9276KiB/s (9499kB/s-9499kB/s), io=544MiB (570MB), run=60001-60001msec