Hallo,
Habe 2x RS 1000 SAS G8 in Benutzung.
Seit 2 Tagen ist die storage performance von einer Maschine extrem schlecht geworden.
Read benchmark Server 1 (OK)
Code
fio --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=8k --numjobs=8 --size=1G --runtime=120 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 8 processes
Jobs: 1 (f=1): [R(1),_(7)][100.0%][r=29.1MiB/s][r=3719 IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=8): err= 0: pid=27849: Wed Jun 24 20:06:25 2020
read: IOPS=21.4k, BW=167MiB/s (175MB/s)(8192MiB/49031msec)
slat (usec): min=4, max=43730, avg=19.61, stdev=120.59
clat (nsec): min=1391, max=231873k, avg=322247.72, stdev=1403505.24
lat (usec): min=62, max=231882, avg=343.10, stdev=1415.74
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 106],
| 30.00th=[ 117], 40.00th=[ 129], 50.00th=[ 143], 60.00th=[ 163],
| 70.00th=[ 194], 80.00th=[ 247], 90.00th=[ 383], 95.00th=[ 537],
| 99.00th=[ 5604], 99.50th=[ 8586], 99.90th=[17695], 99.95th=[23200],
| 99.99th=[41157]
bw ( KiB/s): min= 1936, max=39392, per=13.34%, avg=22816.53, stdev=6711.12, samples=726
iops : min= 242, max= 4924, avg=2852.00, stdev=838.87, samples=726
lat (usec) : 2=0.08%, 4=0.46%, 10=0.03%, 20=0.04%, 50=0.27%
lat (usec) : 100=13.74%, 250=65.68%, 500=13.85%, 750=2.91%, 1000=0.69%
lat (msec) : 2=0.67%, 4=0.33%, 10=0.92%, 20=0.28%, 50=0.07%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.96%, sys=6.96%, ctx=1126028, majf=0, minf=116
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=8192MiB (8590MB), run=49031-49031msec
Disk stats (read/write):
sda: ios=1049015/69, merge=31/71, ticks=309156/522, in_queue=298884, util=98.83%
Display More
Gleicher benchmark auf Server 2 (LAAAAAANNNNGGSAAAAAM)
Code
fio --name=seqread --rw=read --direct=1 --ioengine=libaio --bs=8k --numjobs=8 --size=1G --runtime=120 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
...
fio-3.12
Starting 8 processes
Jobs: 8 (f=8): [R(8)][100.0%][r=158MiB/s][r=20.2k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=8): err= 0: pid=417: Wed Jun 24 20:16:51 2020
read: IOPS=458, BW=3667KiB/s (3755kB/s)(430MiB/120005msec)
slat (usec): min=4, max=6617, avg=19.14, stdev=41.08
clat (nsec): min=1676, max=4748.1M, avg=17425246.93, stdev=180498783.35
lat (usec): min=71, max=4748.2k, avg=17445.44, stdev=180501.38
clat percentiles (usec):
| 1.00th=[ 77], 5.00th=[ 93], 10.00th=[ 101],
| 20.00th=[ 114], 30.00th=[ 125], 40.00th=[ 139],
| 50.00th=[ 157], 60.00th=[ 188], 70.00th=[ 247],
| 80.00th=[ 461], 90.00th=[ 3785], 95.00th=[ 13435],
| 99.00th=[ 278922], 99.50th=[1350566], 99.90th=[2835350],
| 99.95th=[3472884], 99.99th=[4211082]
bw ( KiB/s): min= 15, max=42784, per=29.08%, avg=1066.45, stdev=4044.71, samples=776
iops : min= 1, max= 5348, avg=133.24, stdev=505.60, samples=776
lat (usec) : 2=0.02%, 4=0.14%, 10=0.01%, 20=0.02%, 50=0.12%
lat (usec) : 100=9.18%, 250=60.89%, 500=10.27%, 750=2.55%, 1000=2.03%
lat (msec) : 2=3.02%, 4=1.95%, 10=3.49%, 20=2.60%, 50=1.70%
lat (msec) : 100=0.57%, 250=0.39%, 500=0.29%, 750=0.11%, 1000=0.06%
cpu : usr=0.06%, sys=0.15%, ctx=57048, majf=0, minf=128
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=55012,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3667KiB/s (3755kB/s), 3667KiB/s-3667KiB/s (3755kB/s-3755kB/s), io=430MiB (451MB), run=120005-120005msec
Disk stats (read/write):
sda: ios=52802/64, merge=0/75, ticks=1331698/105566, in_queue=609664, util=45.36%
Display More
Habe schon fstrim und storage optimization laufen lassen. Hat aber nichts geholfen
Gruß,
mo