Bonjour Alain,

Pour la réunion du 11 avril du gull de Luxembourg, ce ne m'est pas possible de présenter un travail abouti sur bcache (mnémotechnique des commandes, optimisation avec le strip choisi pour le RAID, benchmark réel avec fio). En particulier, je n'avais jamais été capable d'expliquer certaines mesures de performances obtenues avec l'utilitaire fio, alors que c'est le coeur du sujet. Il me faut donc consacrer de nouveau deux ou trois jours pour remonter une machine de test.

Deux options s'offrent à nous:

  1. reporter à plus tard cette présentation;
  2. accepter cette présentation "en l'état", dans un esprit d'entre-aide et collaboratif, basée sur mes notes de l'époque, avec de possibles trous de mémoire, c'est à dire un travail partiel sans garantie hélas d'exactitude.

Je te laisse la décision et me réjouis de vous retrouver le 11 avril, quel que soit le sujet de la réunion.

Bien amicalement,

Frédéric.

--
f.dumas@ellis.siteparc.fr



Début du message réexpédié :

De: Frédéric Dumas <f.dumas@ellis.siteparc.fr>
Objet: Bcache on Dell R710 + P3700 - Benchmark
Date: 18 novembre 2021 à 17:24:53 UTC+2
À: Brent Frère <Brent.Frere@AbilIT.eu>
Cc: Michał Rokitiański <michal.rokitianski@softq.pl>


Here are some benchmark results on bcache running on Dell R710 + PERC H700 + P3700 NVMe.




- ! - Setup



Backend device: 6 x HDD SATA 3Gbps in RAID6 hardware (PERC H700 aka LSI MegaRAID SAS 9260-8i)
               HDD sector size = 512 bytes
               RAID6 stride = stripe element size = 1024KB
               RAID6 stripe width = 4096KB
               bcache --data-offset 8192 (sectors)

Cache device  : 2 x P3700 NVme in RAID1 software (md)


Configuration of bcache:

Sequential reads through bcache # echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
Writeback by bcache  # echo writeback > /sys/block/bcache0/bcache/cache_mode


Hints:


# smartctl -iA -d megaraid,{0,5} /dev/sda

gives access to the smart values of the disks running behind the hardware RAID controller.


# make-bcache --data-offset 8192 --bucket 2048k --writeback --bdev /dev/sda4 --cache /dev/md3700

attaches the cache device to the backend device and registers the /dev/bcach0 device.


#root@softq-pve-710:~# lsblk
NAME           SIZE FSTYPE      TYPE  MOUNTPOINT
sda            7.3T             disk  
├─sda1        1007K             part  
├─sda2         512M vfat        part  
├─sda3        31.5G LVM2_member part  
│ └─pve-root   7.8G ext4        lvm   /
└─sda4         7.2T bcache      part  
 └─bcache0    7.2T             disk  
sr0           1024M             rom   
nvme0n1      745.2G             disk  
└─md3700     745.1G bcache      raid1
 └─bcache0    7.2T             disk  
nvme1n1      745.2G             disk  
└─md3700     745.1G bcache      raid1
 └─bcache0    7.2T             disk  




- 2 - Summary



Bcache delivers a x11 performance boost on 64KB random read/writes. It delivers a sustained x2 performance boost on 64KB chunks sequential reads; a modest result, as these reads are already accelerated by the hardware raid controller's read-ahead policy. Bcache delivers no performance boost at all on fully random 64KB reads, as they are spread over a very large 7.2TB space. Likely, such a test should be completed on a few dozen GB large file, not on an entire volume.


Sequential reads without bcache (adaptative read ahead on hardware raid controller)

      1.1       ~655MiB/s        ~10.5k IOPS

Sequential reads with bcache (adaptative read ahead on hardware raid controller)

      1.2       ~1303MiB/s       ~20.9k IOPS


Random reads without bcache (adaptative read ahead on hardware raid controller)

      2.1       ~45.5MiB/s       ~728 IOPS

Random reads with bcache (adaptative read ahead on hardware raid controller)

      2.2       ~45.9MiB/s       ~733 IOPS


Random read/writes without bcache (adaptative read ahead on hardware raid controller)

      3.1       ~6.0MiB/s        ~93 IOPS

Random read/writes with bcache (adaptative read ahead on hardware raid controller)

      3.2       ~67.9MiB/s (R)   ~1086 IOPS
      3.2       ~68.1MiB/s (W)   ~1086 IOPS


Once the last read/write test was completed, it took several minutes to bcache to write the data back to the RAID6 array.



- 3 - Full results



Source:
https://docs.oracle.com/en-us/iaas/Content/Block/References/samplefiocommandslinux.htm



1.1 - Throughput Performance Test - sequential reads - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITHOUT BCACHE



root@softq-pve-710:~# fio --filename=/dev/sda4 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1009110: Thu Nov 18 12:14:25 2021
 read: IOPS=10.5k, BW=655MiB/s (687MB/s)(76.7GiB/120012msec)
   slat (usec): min=5, max=253, avg=20.03, stdev= 5.96
   clat (usec): min=106, max=232703, avg=12192.72, stdev=10140.66
    lat (usec): min=122, max=232718, avg=12213.20, stdev=10140.72
   clat percentiles (usec):
    |  1.00th=[   457],  5.00th=[  1401], 10.00th=[  2311], 20.00th=[  3851],
    | 30.00th=[  5211], 40.00th=[  7570], 50.00th=[ 10028], 60.00th=[ 12911],
    | 70.00th=[ 16057], 80.00th=[ 19530], 90.00th=[ 24249], 95.00th=[ 29230],
    | 99.00th=[ 41681], 99.50th=[ 48497], 99.90th=[ 84411], 99.95th=[125305],
    | 99.99th=[204473]
  bw (  KiB/s): min=287872, max=1654656, per=100.00%, avg=671072.94, stdev=31210.13, samples=956
  iops        : min= 4498, max=25854, avg=10485.52, stdev=487.66, samples=956
 lat (usec)   : 250=0.01%, 500=1.32%, 750=0.95%, 1000=1.11%
 lat (msec)   : 2=4.66%, 4=12.84%, 10=28.89%, 20=31.69%, 50=18.12%
 lat (msec)   : 100=0.35%, 250=0.08%
 cpu          : usr=2.95%, sys=10.94%, ctx=1069436, majf=0, minf=2128
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=1257420,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=655MiB/s (687MB/s), 655MiB/s-655MiB/s (687MB/s-687MB/s), io=76.7GiB (82.4GB), run=120012-120012msec

Disk stats (read/write):
 sda: ios=1255705/246, merge=254/64, ticks=15282727/6083, in_queue=15288810, util=99.99%




1.2 - Throughput Performance Test - sequential reads - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITH BCACHE



root@softq-pve-710:~# fio --filename=/dev/bcache0 --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1027176: Thu Nov 18 14:42:06 2021
 read: IOPS=20.9k, BW=1303MiB/s (1367MB/s)(153GiB/120038msec)
   slat (usec): min=7, max=39687, avg=138.05, stdev=199.59
   clat (usec): min=54, max=755004, avg=5996.18, stdev=11372.86
    lat (usec): min=74, max=755048, avg=6134.93, stdev=11369.60
   clat percentiles (usec):
    |  1.00th=[   416],  5.00th=[  2442], 10.00th=[  3261], 20.00th=[  3884],
    | 30.00th=[  4228], 40.00th=[  4555], 50.00th=[  4817], 60.00th=[  5145],
    | 70.00th=[  5538], 80.00th=[  5997], 90.00th=[  6915], 95.00th=[  9896],
    | 99.00th=[ 34866], 99.50th=[ 44827], 99.90th=[113771], 99.95th=[261096],
    | 99.99th=[541066]
  bw (  MiB/s): min=   48, max= 1701, per=100.00%, avg=1307.64, stdev=109.03, samples=956
  iops        : min=  768, max=27222, avg=20921.69, stdev=1744.45, samples=956
 lat (usec)   : 100=0.01%, 250=0.02%, 500=1.66%, 750=1.04%, 1000=0.30%
 lat (msec)   : 2=0.98%, 4=19.10%, 10=71.97%, 20=2.18%, 50=2.37%
 lat (msec)   : 100=0.25%, 250=0.06%, 500=0.05%, 750=0.01%, 1000=0.01%
 cpu          : usr=3.24%, sys=36.24%, ctx=3406755, majf=0, minf=34793
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=2503357,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=1303MiB/s (1367MB/s), 1303MiB/s-1303MiB/s (1367MB/s-1367MB/s), io=153GiB (164GB), run=120038-120038msec

Disk stats (read/write):
   bcache0: ios=2502109/0, merge=0/0, ticks=7149936/0, in_queue=7149936, util=99.86%, aggrios=1262209/1106438, aggrmerge=4/54, aggrticks=2461402/1258470, aggrin_queue=3719872, aggrutil=99.02%
 sda: ios=2059550/295, merge=9/108, ticks=4031424/35305, in_queue=4066729, util=99.02%
   md3700: ios=464868/2212581, merge=0/0, ticks=891380/2481636, in_queue=3373016, util=95.12%, aggrios=232423/2114558, aggrmerge=11/101552, aggrticks=398099/1256745, aggrin_queue=1654844, aggrutil=95.05%
 nvme0n1: ios=238263/2115522, merge=12/100588, ticks=410475/1286394, in_queue=1696870, util=95.05%
 nvme1n1: ios=226583/2113594, merge=10/102516, ticks=385723/1227097, in_queue=1612819, util=94.99%




2.1 - Throughput Performance Test - random reads - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITHOUT BCACHE



root@softq-pve-710:~# fio --filename=/dev/sda4 --direct=1 --rw=randread --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1010415: Thu Nov 18 12:24:50 2021
 read: IOPS=728, BW=45.5MiB/s (47.7MB/s)(5474MiB/120290msec)
   slat (usec): min=9, max=350, avg=23.57, stdev= 3.17
   clat (usec): min=164, max=2280.6k, avg=175684.34, stdev=159256.80
    lat (usec): min=187, max=2280.6k, avg=175708.55, stdev=159256.79
   clat percentiles (msec):
    |  1.00th=[   25],  5.00th=[   31], 10.00th=[   38], 20.00th=[   54],
    | 30.00th=[   74], 40.00th=[   99], 50.00th=[  127], 60.00th=[  163],
    | 70.00th=[  207], 80.00th=[  271], 90.00th=[  380], 95.00th=[  489],
    | 99.00th=[  760], 99.50th=[  894], 99.90th=[ 1183], 99.95th=[ 1351],
    | 99.99th=[ 1636]
  bw (  KiB/s): min=33664, max=58368, per=100.00%, avg=46646.60, stdev=1045.77, samples=960
  iops        : min=  526, max=  912, avg=728.84, stdev=16.35, samples=960
 lat (usec)   : 250=0.01%
 lat (msec)   : 20=0.14%, 50=17.84%, 100=22.94%, 250=36.48%, 500=17.98%
 lat (msec)   : 750=3.58%, 1000=0.77%, 2000=0.27%, >=2000=0.01%
 cpu          : usr=0.36%, sys=1.04%, ctx=86189, majf=0, minf=2109
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=87587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=5474MiB (5740MB), run=120290-120290msec

Disk stats (read/write):
 sda: ios=87540/205, merge=0/62, ticks=15349921/28134, in_queue=15378056, util=99.99%




2.2 - Throughput Performance Test - random reads - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITH BCACHE



root@softq-pve-710:~# fio --filename=/dev/bcache0 --direct=1 --rw=randread --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1030606: Thu Nov 18 14:55:16 2021
 read: IOPS=733, BW=45.9MiB/s (48.1MB/s)(5522MiB/120391msec)
   slat (usec): min=21, max=6740, avg=58.67, stdev=90.27
   clat (usec): min=234, max=2413.0k, avg=174256.84, stdev=157810.22
    lat (usec): min=273, max=2413.1k, avg=174316.24, stdev=157809.87
   clat percentiles (msec):
    |  1.00th=[   23],  5.00th=[   31], 10.00th=[   37], 20.00th=[   54],
    | 30.00th=[   73], 40.00th=[   97], 50.00th=[  127], 60.00th=[  163],
    | 70.00th=[  207], 80.00th=[  268], 90.00th=[  372], 95.00th=[  481],
    | 99.00th=[  760], 99.50th=[  877], 99.90th=[ 1183], 99.95th=[ 1334],
    | 99.99th=[ 1687]
  bw (  KiB/s): min=33328, max=57955, per=100.00%, avg=47075.48, stdev=1075.87, samples=960
  iops        : min=  520, max=  905, avg=734.54, stdev=16.81, samples=960
 lat (usec)   : 250=0.01%, 500=0.44%, 750=0.06%, 1000=0.01%
 lat (msec)   : 2=0.01%, 4=0.01%, 20=0.18%, 50=17.58%, 100=22.79%
 lat (msec)   : 250=36.41%, 500=18.09%, 750=3.40%, 1000=0.79%, 2000=0.23%
 lat (msec)   : >=2000=0.01%
 cpu          : usr=0.31%, sys=1.98%, ctx=87526, majf=0, minf=2105
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=88357,0,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=45.9MiB/s (48.1MB/s), 45.9MiB/s-45.9MiB/s (48.1MB/s-48.1MB/s), io=5522MiB (5791MB), run=120391-120391msec

Disk stats (read/write):
   bcache0: ios=88357/0, merge=0/0, ticks=15365196/0, in_queue=15365196, util=100.00%, aggrios=44195/44624, aggrmerge=0/26, aggrticks=7679901/20858, aggrin_queue=7700759, aggrutil=100.00%
 sda: ios=87899/210, merge=0/53, ticks=15359634/28724, in_queue=15388358, util=100.00%
   md3700: ios=492/89038, merge=0/0, ticks=168/12992, in_queue=13160, util=98.35%, aggrios=246/89109, aggrmerge=0/668, aggrticks=98/9289, aggrin_queue=9388, aggrutil=98.16%
 nvme0n1: ios=489/89108, merge=0/670, ticks=196/8879, in_queue=9075, util=98.16%
 nvme1n1: ios=3/89111, merge=0/667, ticks=1/9700, in_queue=9702, util=98.15%




3.1 - Throughput Performance Test - random read/writes - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITHOUT BCACHE



root@softq-pve-710:~# fio --filename=/dev/sda4 --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1011093: Thu Nov 18 12:30:15 2021
 read: IOPS=93, BW=5998KiB/s (6142kB/s)(707MiB/120737msec)
   slat (usec): min=10, max=224, avg=25.81, stdev= 2.94
   clat (msec): min=11, max=4247, avg=347.85, stdev=434.68
    lat (msec): min=11, max=4247, avg=347.87, stdev=434.68
   clat percentiles (msec):
    |  1.00th=[   26],  5.00th=[   34], 10.00th=[   41], 20.00th=[   55],
    | 30.00th=[   80], 40.00th=[  118], 50.00th=[  169], 60.00th=[  243],
    | 70.00th=[  363], 80.00th=[  592], 90.00th=[  927], 95.00th=[ 1234],
    | 99.00th=[ 1989], 99.50th=[ 2400], 99.90th=[ 3205], 99.95th=[ 3406],
    | 99.99th=[ 4178]
  bw (  KiB/s): min= 1152, max=12928, per=100.00%, avg=6014.06, stdev=506.65, samples=960
  iops        : min=   18, max=  202, avg=93.97, stdev= 7.92, samples=960
 write: IOPS=96, BW=6204KiB/s (6353kB/s)(732MiB/120737msec); 0 zone resets
   slat (usec): min=13, max=294, avg=31.81, stdev= 4.31
   clat (msec): min=114, max=5783, avg=983.56, stdev=411.47
    lat (msec): min=114, max=5783, avg=983.60, stdev=411.47
   clat percentiles (msec):
    |  1.00th=[  376],  5.00th=[  510], 10.00th=[  584], 20.00th=[  676],
    | 30.00th=[  760], 40.00th=[  827], 50.00th=[  894], 60.00th=[  969],
    | 70.00th=[ 1083], 80.00th=[ 1217], 90.00th=[ 1502], 95.00th=[ 1770],
    | 99.00th=[ 2433], 99.50th=[ 2702], 99.90th=[ 3440], 99.95th=[ 3943],
    | 99.99th=[ 5269]
  bw (  KiB/s): min= 1792, max=11520, per=99.87%, avg=6196.98, stdev=410.22, samples=960
  iops        : min=   28, max=  180, avg=96.82, stdev= 6.41, samples=960
 lat (msec)   : 20=0.07%, 50=8.43%, 100=9.20%, 250=12.28%, 500=9.98%
 lat (msec)   : 750=16.81%, 1000=20.10%, 2000=21.25%, >=2000=1.89%
 cpu          : usr=0.11%, sys=0.34%, ctx=22938, majf=0, minf=71
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=11315,11704,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=5998KiB/s (6142kB/s), 5998KiB/s-5998KiB/s (6142kB/s-6142kB/s), io=707MiB (742MB), run=120737-120737msec
 WRITE: bw=6204KiB/s (6353kB/s), 6204KiB/s-6204KiB/s (6353kB/s-6353kB/s), io=732MiB (767MB), run=120737-120737msec

Disk stats (read/write):
 sda: ios=11370/11806, merge=0/45, ticks=3916382/11605103, in_queue=15521484, util=99.99%




3.2 - Throughput Performance Test - random read/writes - WITH ADAPTIVE READ AHEAD HARDWARE CACHE + WITH BCACHE



root@softq-pve-710:~# fio --filename=/dev/bcache0 --direct=1 --rw=randrw --bs=64k --ioengine=libaio --iodepth=32 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1



throughput-test-job: (groupid=0, jobs=4): err= 0: pid=1031738: Thu Nov 18 15:04:05 2021
 read: IOPS=1086, BW=67.9MiB/s (71.2MB/s)(8161MiB/120229msec)
   slat (usec): min=15, max=86127, avg=51.75, stdev=568.52
   clat (usec): min=188, max=2007.7k, avg=116178.43, stdev=151824.06
    lat (usec): min=242, max=2007.7k, avg=116230.79, stdev=151836.54
   clat percentiles (usec):
    |  1.00th=[    441],  5.00th=[    652], 10.00th=[    832],
    | 20.00th=[   1385], 30.00th=[   2999], 40.00th=[  36439],
    | 50.00th=[  61604], 60.00th=[  96994], 70.00th=[ 141558],
    | 80.00th=[ 204473], 90.00th=[ 308282], 95.00th=[ 413139],
    | 99.00th=[ 683672], 99.50th=[ 801113], 99.90th=[1082131],
    | 99.95th=[1216349], 99.99th=[1535116]
  bw (  KiB/s): min=30834, max=1256048, per=100.00%, avg=69675.99, stdev=38920.70, samples=957
  iops        : min=  481, max=19625, avg=1087.75, stdev=608.16, samples=957
 write: IOPS=1089, BW=68.1MiB/s (71.4MB/s)(8190MiB/120229msec); 0 zone resets
   slat (usec): min=14, max=50364, avg=47.67, stdev=339.27
   clat (usec): min=11, max=165668, avg=1454.15, stdev=2560.77
    lat (usec): min=74, max=165915, avg=1502.47, stdev=2590.82
   clat percentiles (usec):
    |  1.00th=[   71],  5.00th=[   73], 10.00th=[   80], 20.00th=[  119],
    | 30.00th=[  135], 40.00th=[  139], 50.00th=[  145], 60.00th=[  169],
    | 70.00th=[ 1221], 80.00th=[ 3359], 90.00th=[ 5080], 95.00th=[ 6194],
    | 99.00th=[ 8586], 99.50th=[ 9503], 99.90th=[14484], 99.95th=[21627],
    | 99.99th=[26608]
  bw (  KiB/s): min=23655, max=1247613, per=100.00%, avg=69995.69, stdev=39278.35, samples=957
  iops        : min=  368, max=19493, avg=1092.74, stdev=613.74, samples=957
 lat (usec)   : 20=0.01%, 50=0.01%, 100=7.71%, 250=25.16%, 500=1.76%
 lat (usec)   : 750=3.40%, 1000=3.65%
 lat (msec)   : 2=7.17%, 4=9.16%, 10=8.75%, 20=0.24%, 50=5.96%
 lat (msec)   : 100=7.50%, 250=12.11%, 500=5.94%, 750=1.17%, 1000=0.24%
 lat (msec)   : 2000=0.08%, >=2000=0.01%
 cpu          : usr=0.76%, sys=3.07%, ctx=225226, majf=1, minf=10344
 IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
    submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    issued rwts: total=130582,131041,0,0 short=0,0,0,0 dropped=0,0,0,0
    latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  READ: bw=67.9MiB/s (71.2MB/s), 67.9MiB/s-67.9MiB/s (71.2MB/s-71.2MB/s), io=8161MiB (8558MB), run=120229-120229msec
 WRITE: bw=68.1MiB/s (71.4MB/s), 68.1MiB/s-68.1MiB/s (71.4MB/s-71.4MB/s), io=8190MiB (8588MB), run=120229-120229msec

Disk stats (read/write):
   bcache0: ios=130619/131041, merge=0/0, ticks=15128128/174484, in_queue=15302612, util=100.00%, aggrios=65454/110552, aggrmerge=0/34, aggrticks=7565070/131037, aggrin_queue=7696107, aggrutil=99.57%
 sda: ios=86501/300, merge=0/68, ticks=15072824/78563, in_queue=15151387, util=99.57%
   md3700: ios=44408/220805, merge=0/0, ticks=57316/183512, in_queue=240828, util=98.53%, aggrios=22202/221689, aggrmerge=0/548, aggrticks=28755/138402, aggrin_queue=167156, aggrutil=98.41%
 nvme0n1: ios=22193/221688, merge=0/549, ticks=28763/138527, in_queue=167289, util=98.39%
 nvme1n1: ios=22212/221690, merge=0/547, ticks=28747/138277, in_queue=167024, util=98.41%



--
Frédéric Dumas
f.dumas@ellis.siteparc.fr