Re: tools/virtiofs: Multi threading seems to hurt performance

[Vivek Goyal]

On Tue, Sep 22, 2020 at 12:09:46PM +0100, Dr. David Alan Gilbert wrote:
> 
> Do you have the numbers for:
>    epool
>    epool thread-pool-size=1
>    spool

Hi David,

Ok, I re-ran my numbers again after upgrading to latest qemu and also
upgraded host kernel to latest upstream. Apart from comparing I epool,
spool and 1Thread, I also ran their numa variants. That is I launched
qemu and virtiofsd on node 0 of machine (numactl --cpunodebind=0).

Results are kind of mixed. Here are my takeaways.

- Running on same numa node improves performance overall for exclusive,
  shared and exclusive-1T mode.

- In general both shared pool and exclusive-1T mode seem to perform
  better than exclusive mode, except for the case of randwrite-libaio.
  In some cases (seqread-libaio, seqwrite-libaio, seqwrite-libaio-multi)
  exclusive pool performs better than exclusive-1T.

- Looks like in some cases exclusive-1T performs better than shared
  pool. (randwrite-libaio, randwrite-psync-multi, seqwrite-psync-multi,
  seqwrite-psync, seqread-libaio-multi, seqread-psync-multi)


Overall, I feel that both exlusive-1T and shared perform better than
exclusive pool. Results between exclusive-1T and shared pool are mixed.
It seems like in many cases exclusve-1T performs better. I would say
that moving to "shared" pool seems like a reasonable option.

Thanks
Vivek

NAME                    WORKLOAD                Bandwidth       IOPS            
vtfs-none-epool         seqread-psync           38(MiB/s)       9967            
vtfs-none-epool-1T      seqread-psync           66(MiB/s)       16k             
vtfs-none-spool         seqread-psync           67(MiB/s)       16k             
vtfs-none-epool-numa    seqread-psync           48(MiB/s)       12k             
vtfs-none-epool-1T-numa seqread-psync           74(MiB/s)       18k             
vtfs-none-spool-numa    seqread-psync           74(MiB/s)       18k             

vtfs-none-epool         seqread-psync-multi     204(MiB/s)      51k             
vtfs-none-epool-1T      seqread-psync-multi     325(MiB/s)      81k             
vtfs-none-spool         seqread-psync-multi     271(MiB/s)      67k             
vtfs-none-epool-numa    seqread-psync-multi     253(MiB/s)      63k             
vtfs-none-epool-1T-numa seqread-psync-multi     349(MiB/s)      87k             
vtfs-none-spool-numa    seqread-psync-multi     301(MiB/s)      75k             

vtfs-none-epool         seqread-libaio          301(MiB/s)      75k             
vtfs-none-epool-1T      seqread-libaio          273(MiB/s)      68k             
vtfs-none-spool         seqread-libaio          334(MiB/s)      83k             
vtfs-none-epool-numa    seqread-libaio          315(MiB/s)      78k             
vtfs-none-epool-1T-numa seqread-libaio          326(MiB/s)      81k             
vtfs-none-spool-numa    seqread-libaio          335(MiB/s)      83k             

vtfs-none-epool         seqread-libaio-multi    202(MiB/s)      50k             
vtfs-none-epool-1T      seqread-libaio-multi    308(MiB/s)      77k             
vtfs-none-spool         seqread-libaio-multi    247(MiB/s)      61k             
vtfs-none-epool-numa    seqread-libaio-multi    238(MiB/s)      59k             
vtfs-none-epool-1T-numa seqread-libaio-multi    307(MiB/s)      76k             
vtfs-none-spool-numa    seqread-libaio-multi    269(MiB/s)      67k             

vtfs-none-epool         randread-psync          41(MiB/s)       10k             
vtfs-none-epool-1T      randread-psync          67(MiB/s)       16k             
vtfs-none-spool         randread-psync          64(MiB/s)       16k             
vtfs-none-epool-numa    randread-psync          48(MiB/s)       12k             
vtfs-none-epool-1T-numa randread-psync          73(MiB/s)       18k             
vtfs-none-spool-numa    randread-psync          72(MiB/s)       18k             

vtfs-none-epool         randread-psync-multi    207(MiB/s)      51k             
vtfs-none-epool-1T      randread-psync-multi    313(MiB/s)      78k             
vtfs-none-spool         randread-psync-multi    265(MiB/s)      66k             
vtfs-none-epool-numa    randread-psync-multi    253(MiB/s)      63k             
vtfs-none-epool-1T-numa randread-psync-multi    340(MiB/s)      85k             
vtfs-none-spool-numa    randread-psync-multi    305(MiB/s)      76k             

vtfs-none-epool         randread-libaio         305(MiB/s)      76k             
vtfs-none-epool-1T      randread-libaio         308(MiB/s)      77k             
vtfs-none-spool         randread-libaio         329(MiB/s)      82k             
vtfs-none-epool-numa    randread-libaio         310(MiB/s)      77k             
vtfs-none-epool-1T-numa randread-libaio         328(MiB/s)      82k             
vtfs-none-spool-numa    randread-libaio         339(MiB/s)      84k             

vtfs-none-epool         randread-libaio-multi   265(MiB/s)      66k             
vtfs-none-epool-1T      randread-libaio-multi   267(MiB/s)      66k             
vtfs-none-spool         randread-libaio-multi   269(MiB/s)      67k             
vtfs-none-epool-numa    randread-libaio-multi   314(MiB/s)      78k             
vtfs-none-epool-1T-numa randread-libaio-multi   319(MiB/s)      79k             
vtfs-none-spool-numa    randread-libaio-multi   318(MiB/s)      79k             

vtfs-none-epool         seqwrite-psync          36(MiB/s)       9224            
vtfs-none-epool-1T      seqwrite-psync          67(MiB/s)       16k             
vtfs-none-spool         seqwrite-psync          61(MiB/s)       15k             
vtfs-none-epool-numa    seqwrite-psync          44(MiB/s)       11k             
vtfs-none-epool-1T-numa seqwrite-psync          69(MiB/s)       17k             
vtfs-none-spool-numa    seqwrite-psync          68(MiB/s)       17k             

vtfs-none-epool         seqwrite-psync-multi    193(MiB/s)      48k             
vtfs-none-epool-1T      seqwrite-psync-multi    299(MiB/s)      74k             
vtfs-none-spool         seqwrite-psync-multi    240(MiB/s)      60k             
vtfs-none-epool-numa    seqwrite-psync-multi    233(MiB/s)      58k             
vtfs-none-epool-1T-numa seqwrite-psync-multi    358(MiB/s)      89k             
vtfs-none-spool-numa    seqwrite-psync-multi    285(MiB/s)      71k             

vtfs-none-epool         seqwrite-libaio         265(MiB/s)      66k             
vtfs-none-epool-1T      seqwrite-libaio         245(MiB/s)      61k             
vtfs-none-spool         seqwrite-libaio         312(MiB/s)      78k             
vtfs-none-epool-numa    seqwrite-libaio         295(MiB/s)      73k             
vtfs-none-epool-1T-numa seqwrite-libaio         282(MiB/s)      70k             
vtfs-none-spool-numa    seqwrite-libaio         297(MiB/s)      74k             

vtfs-none-epool         seqwrite-libaio-multi   313(MiB/s)      78k             
vtfs-none-epool-1T      seqwrite-libaio-multi   299(MiB/s)      74k             
vtfs-none-spool         seqwrite-libaio-multi   315(MiB/s)      78k             
vtfs-none-epool-numa    seqwrite-libaio-multi   318(MiB/s)      79k             
vtfs-none-epool-1T-numa seqwrite-libaio-multi   410(MiB/s)      102k            
vtfs-none-spool-numa    seqwrite-libaio-multi   378(MiB/s)      94k             

vtfs-none-epool         randwrite-psync         33(MiB/s)       8629            
vtfs-none-epool-1T      randwrite-psync         61(MiB/s)       15k             
vtfs-none-spool         randwrite-psync         63(MiB/s)       15k             
vtfs-none-epool-numa    randwrite-psync         49(MiB/s)       12k             
vtfs-none-epool-1T-numa randwrite-psync         68(MiB/s)       17k             
vtfs-none-spool-numa    randwrite-psync         66(MiB/s)       16k             

vtfs-none-epool         randwrite-psync-multi   186(MiB/s)      46k             
vtfs-none-epool-1T      randwrite-psync-multi   300(MiB/s)      75k             
vtfs-none-spool         randwrite-psync-multi   233(MiB/s)      58k             
vtfs-none-epool-numa    randwrite-psync-multi   235(MiB/s)      58k             
vtfs-none-epool-1T-numa randwrite-psync-multi   355(MiB/s)      88k             
vtfs-none-spool-numa    randwrite-psync-multi   266(MiB/s)      66k             

vtfs-none-epool         randwrite-libaio        289(MiB/s)      72k             
vtfs-none-epool-1T      randwrite-libaio        284(MiB/s)      71k             
vtfs-none-spool         randwrite-libaio        278(MiB/s)      69k             
vtfs-none-epool-numa    randwrite-libaio        292(MiB/s)      73k             
vtfs-none-epool-1T-numa randwrite-libaio        294(MiB/s)      73k             
vtfs-none-spool-numa    randwrite-libaio        290(MiB/s)      72k             

vtfs-none-epool         randwrite-libaio-multi  317(MiB/s)      79k             
vtfs-none-epool-1T      randwrite-libaio-multi  323(MiB/s)      80k             
vtfs-none-spool         randwrite-libaio-multi  330(MiB/s)      82k             
vtfs-none-epool-numa    randwrite-libaio-multi  315(MiB/s)      78k             
vtfs-none-epool-1T-numa randwrite-libaio-multi  409(MiB/s)      102k            
vtfs-none-spool-numa    randwrite-libaio-multi  384(MiB/s)      96k