Re: virtiofs vs 9p performance(Re: tools/virtiofs: Multi threading seems to hurt performance)

[Vivek Goyal]

On Fri, Sep 25, 2020 at 01:41:39PM +0100, Dr. David Alan Gilbert wrote:

[..]
> So I'm sitll beating 9p; the thread-pool-size=1 seems to be great for
> read performance here.
> 

Hi Dave,

I spent some time making changes to virtiofs-tests so that I can test
a mix of random read and random write workload. That testsuite runs
a workload 3 times and reports the average. So I like to use it to
reduce run to run variation effect.

So I ran following to mimic carlos's workload.

$ ./run-fio-test.sh test -direct=1 -c <test-dir> fio-jobs/randrw-psync.job >
testresults.txt

$ ./parse-fio-results.sh testresults.txt

I am using a SSD at the host to back these files. Option "-c" always
creates new files for testing.

Following are my results in various configurations. Used cache=mmap mode
for 9p and cache=auto (and cache=none) modes for virtiofs. Also tested
9p default as well as msize=16m. Tested virtiofs both with exclusive
as well as shared thread pool.

NAME                    WORKLOAD                Bandwidth       IOPS            
9p-mmap-randrw          randrw-psync            42.8mb/14.3mb   10.7k/3666      
9p-mmap-msize16m        randrw-psync            42.8mb/14.3mb   10.7k/3674      
vtfs-auto-ex-randrw     randrw-psync            27.8mb/9547kb   7136/2386       
vtfs-auto-sh-randrw     randrw-psync            43.3mb/14.4mb   10.8k/3709      
vtfs-none-sh-randrw     randrw-psync            54.1mb/18.1mb   13.5k/4649      


- Increasing msize to 16m did not help with performance for this workload.
- virtiofs exclusive thread pool ("ex"), is slower than 9p.
- virtiofs shared thread pool ("sh"), matches the performance of 9p.
- virtiofs cache=none mode is faster than cache=auto mode for this
  workload.

Carlos, I am looking at more ways to optimize it further for virtiofs.
In the mean time I think switching to "shared" thread pool should
bring you very close to 9p in your setup I think.

Thanks
Vivek