Re: tools/virtiofs: Multi threading seems to hurt performance

[Vivek Goyal]

On Mon, Sep 21, 2020 at 09:39:23AM +0100, Stefan Hajnoczi wrote:
> On Fri, Sep 18, 2020 at 05:34:36PM -0400, Vivek Goyal wrote:
> > And here are the comparision results. To me it seems that by default
> > we should switch to 1 thread (Till we can figure out how to make
> > multi thread performance better even when single process is doing
> > I/O in client).
> 
> Let's understand the reason before making changes.
> 
> Questions:
>  * Is "1-thread" --thread-pool-size=1?

Yes.

>  * Was DAX enabled?

No.

>  * How does cache=none perform?

I just ran random read workload with cache=none.

cache-none              randread-psync          45(MiB/s)       11k             
cache-none-1-thread     randread-psync          63(MiB/s)       15k

With 1 thread it offers more IOPS.

>  * Does commenting out vu_queue_get_avail_bytes() + fuse_log("%s:
>    Queue %d gave evalue: %zx available: in: %u out: %u\n") in
>    fv_queue_thread help?

Will try that.

>  * How do the kvm_stat vmexit counters compare?

This should be same, isn't it. Changing number of threads serving should
not change number of vmexits?

>  * How does host mpstat -P ALL compare?

Never used mpstat. Will try running it and see if I can get something
meaningful.

>  * How does host perf record -a compare?

Will try it. I feel this might be too big and too verbose to get
something meaningful.

>  * Does the Rust virtiofsd show the same pattern (it doesn't use glib
>    thread pools)?

No idea. Never tried rust implementation of virtiofsd.

But I suepct it has to do with thread pool implementation and possibly
extra cacheline bouncing.

Thanks
Vivek