Re: [PATCH v2 0/4] Support dynamic MSI-X allocation

[Alex Williamson]

On Mon, 18 Sep 2023 05:45:03 -0400
Jing Liu <jing2.liu@intel.com> wrote:

> Changes since v1:
> - v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html
> - Revise Qemu to QEMU. (Cédric)
> - Add g_free when failure of getting MSI-X irq info. (Cédric)
> - Apply Cédric's Reviewed-by. (Cédric)
> - Use g_autofree to automatically release. (Cédric)
> - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric)
> 
> Changes since RFC v1:
> - RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html
> - Revise the comments. (Alex)
> - Report error of getting irq info and remove the trace of failure
>   case. (Alex, Cédric)
> - Only store dynamic allocation flag as a bool type and test
>   accordingly. (Alex)
> - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex)
> - Change the condition logic in vfio_msix_vector_do_use() that moving
>   the defer_kvm_irq_routing test out and create a common place to update
>   nr_vectors. (Alex)
> - Consolidate the way of MSI-X enabling during device initialization and
>   interrupt restoring that uses fd = -1 trick. Create a function doing
>   that. (Alex)
> 
> Before kernel v6.5, dynamic allocation of MSI-X interrupts was not
> supported. QEMU therefore when allocating a new interrupt, should first
> release all previously allocated interrupts (including disable of MSI-X)
> and re-allocate all interrupts that includes the new one.
> 
> The kernel series [1] adds the support of dynamic MSI-X allocation to
> vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user
> space, that when dynamic MSI-X is supported the flag is cleared.
> 
> This series makes the behavior for VFIO PCI devices when dynamic MSI-X
> allocation is supported. When guest unmasks an interrupt, QEMU can
> directly allocate an interrupt on host for this and has nothing to do
> with the previously allocated ones. Therefore, host only allocates
> interrupts for those unmasked (enabled) interrupts inside guest when
> dynamic MSI-X allocation is supported by device.
> 
> When guests enable MSI-X with all of the vectors masked, QEMU need match
> the state to enable MSI-X with no vector enabled. During migration
> restore, QEMU also need enable MSI-X first in dynamic allocation mode,
> to avoid the guest unused vectors being allocated on host. To
> consolidate them, we use vector 0 with an invalid fd to get MSI-X
> enabled and create a common function for this. This is cleaner than
> setting userspace triggering and immediately release.
> 
> Any feedback is appreciated.
> 
> Jing
> 
> [1] https://lwn.net/Articles/931679/
> 
> Jing Liu (4):
>   vfio/pci: detect the support of dynamic MSI-X allocation
>   vfio/pci: enable vector on dynamic MSI-X allocation
>   vfio/pci: use an invalid fd to enable MSI-X
>   vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation
> 
>  hw/vfio/pci.c        | 121 +++++++++++++++++++++++++++++++++----------
>  hw/vfio/pci.h        |   1 +
>  hw/vfio/trace-events |   2 +-
>  3 files changed, 96 insertions(+), 28 deletions(-)
> 

Some minor comments on 2/ but otherwise:

Reviewed-by: Alex Williamson <alex.williamson@redhat.com>