Re: Issue with VDUSE (QSD vduse-blk export) and vhost-vdpa

[Yongji Xie]

Hi Stefano,

On Wed, Oct 26, 2022 at 5:12 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> Hi Xie,
> I was testing libblkio [1] with QSD vduse-blk export and had some
> issues.
>
> In a nutshell, QSD prints me the following messages when using
> vhost-vdpa to access the device:
>
>   Failed to get vq[0] iova mapping
>   Failed to update vring for vq[0]
>
> This happens only with vhost-vdpa, using virtio-vdpa instead the device
> works fine.
> I'm using Linux v6.0 and QEMU master (commit
> 214a8da23651f2472b296b3293e619fd58d9e212).
>
> I haven't had much time to investigate, I hope to do it next week, but
> maybe it's much faster for you.
>
> I saw that ioctl(VDUSE_IOTLB_GET_FD) in libvduse.c returns -1 (EPERM),
> so IIUC in the kernel vduse_dev_broken() was called, and the device is
> in a broken state.
>
>
> We will use libblkio in QEMU [2] to access vDPA devices via vhost-vdpa.
> But I'm doing these tests without QEMU for now, using an example inside
> the libblkio repo:
>
> # Build libblkio and examples
>     # Fedora/CentOS/RHEL
>     dnf install -y git meson rust cargo python3-docutils rustfmt
>     # Debian/Ubuntu
>     apt-get install -y git meson rustc cargo python3-docutils
>
>     git clone https://gitlab.com/libblkio/libblkio.git
>
>     cd libblkio
>     git checkout v1.1.0
>
>     meson setup build
>     meson compile -C build
>
>
> # On terminal 1
>     modprobe vduse
>     modprobe vhost-vdpa
>
>     qemu-img create -f qcow2 -o preallocation=full /path/to/test.qcow2 1g
>
>     qemu-storage-daemon \
>       --blockdev 
> file,filename=/path/to/test.qcow2,cache.direct=on,aio=native,node-name=file \
>       --blockdev qcow2,file=file,node-name=qcow2 \
>       --object iothread,id=iothread0 \
>       --export 
> vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on,iothread=iothread0
>
>
> # On terminal 2
>     vdpa dev add name vduse0 mgmtdev vduse
>
>     cd libblkio/build
>
>     # blkio-bench executes
>     ./examples/blkio-bench virtio-blk-vhost-vdpa \
>       path=/dev/vhost-vdpa-0 --runtime=5 --readwrite=randread
>
>     # after this step, QSD (running on terminal 1) prints the following 
> messages:
>       Failed to get vq[0] iova mapping
>       Failed to update vring for vq[0]
>
> I don't know if I'm doing something wrong or in libblkio we have some
> issue, but using vdpa-sim-blk works correctly, so maybe there is
> something in vduse that is missing.
>
> Any help or suggestion is welcome :-)
>

I'd like to know whether bio-bench uses the shared memory
(tmpfs/hugetlbfs) as the vdpa memory region. This is what VDUSE needs.

Thanks,
Yongji