Re: [RFC] hw/nvme: Use irqfd to send interrupts

[Stefan Hajnoczi]

On Wed, 27 Jul 2022 at 03:18, Klaus Jensen <its@irrelevant.dk> wrote:
>
> On Jul 21 09:29, Stefan Hajnoczi wrote:
> > On Wed, Jul 20, 2022, 22:36 Jinhao Fan <fanjinhao21s@ict.ac.cn> wrote:
> >
> > > Hi Stefan,
> > >
> > > Thanks for the detailed explanation!
> > >
> > > at 6:21 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >
> > > > Hi Jinhao,
> > > > Thanks for working on this!
> > > >
> > > > irqfd is not necessarily faster than KVM ioctl interrupt injection.
> > > >
> > > > There are at least two non performance reasons for irqfd:
> > > > 1. It avoids QEMU emulation code, which historically was not thread safe
> > > and needed the Big QEMU Lock. IOThreads don't hold the BQL and therefore
> > > cannot safely call the regular interrupt emulation code in QEMU. I think
> > > this is still true today although parts of the code may now be less 
> > > reliant
> > > on the BQL.
> > >
> > > This probably means we need to move to irqfd when iothread support is 
> > > added
> > > in qemu-nvme.
> > >
> >
> > Yes. You can audit the interrupt code but I'm pretty sure there is shared
> > state that needs to be protected by the BQL. So the NVMe emulation code
> > probably needs to use irqfd to avoid the interrupt emulation code.
> >
> >
> > > > 2. The eventfd interface decouples interrupt injection from the KVM
> > > ioctl interface. Vhost kernel and vhost-user device emulation code has no
> > > dependency on KVM thanks to irqfd. They work with any eventfd, including
> > > irqfd.
> > >
> > > This is contrary to our original belief. Klaus once pointed out that irqfd
> > > is KVM specific. I agreed with him since I found irqfd implementation is 
> > > in
> > > virt/kvm/eventfd.c. But irqfd indeed avoids the KVM ioctl call. Could you
> > > elaborate on what “no dependency on KVM” means?
> > >
> >
> > "They work with any eventfd, including irqfd"
> >
> > If you look at the vhost kernel or vhost-user code, you'll see they just
> > signal the eventfd. It doesn't have to be an irqfd.
> >
> > An irqfd is a specific type of eventfd that the KVM kernel module
> > implements to inject interrupts when the eventfd is signaled.
> >
> > By the way, this not only decouples vhost from the KVM kernel module, but
> > also allows QEMU to emulate MSI-X masking via buffering the interrupt in
> > userspace.
> >
> >
>
> The virtio dataplane (iothread support) only works with kvm if I am not
> mistaken, so I guess this is similar to what we want to do here. If we
> dont have KVM, we wont use iothread and we wont use the kvm
> irqchip/irqfd.
>
> Am I understanding this correctly?

I think that is incorrect. QEMU has guest notifier emulation for the
non-KVM (and non-MSI-X PCI) cases. When there is no irqfd support
available, QEMU sets up a regular eventfd and calls
virtio_queue_guest_notifier_read() when it becomes readable.

Stefan