Re: nested-smmuv3 topic for QEMU/libvirt, Nov 2024

[Eric Auger]

Hi Nicolin,

On 11/7/24 21:31, Nicolin Chen wrote:
> Hi Eric,
>
> On Thu, Nov 07, 2024 at 12:11:05PM +0100, Eric Auger wrote:
>> On 11/1/24 05:09, Nicolin Chen wrote:
>>> Hi,
>>>
>>> This is a continued discussion following previous month's:
>>> https://lore.kernel.org/qemu-devel/Zvr%2Fbf7KgLN1cjOl@Asurada-Nvidia/
>>>
>>> Kernel changes are getting closer to merge, as Jason's planning to
>>> take vIOMMU series and smmuv3_nesting series into the iommufd tree:
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/cover.1730313494.git.nicolinc@nvidia.com/
>>> https://lore.kernel.org/all/0-v4-9e99b76f3518+3a8-smmuv3_nesting_jgg@nvidia.com/
>>>
>>> So, I think it's probably a good time to align with each others and
>>> talk about kicking off some QEMU upstream work in the month ahead.
>>>
>>> @Shameer,
>>> Do you have some update on the pluggable smmuv3 module?
>>>
>>> Updates on my side:
>>> 1) I have kept uAPI updated to the latest version and verified too.
>>>    There should be some polishing changes depending on how the basic
>>>    nesting infrastructure would look like from Intel/Duan's work.
>>> 2) I got some help from NVIDIA folks for the libvirt task. And they
>>>    have done some drafting and are now verifying the PCI topology
>>>    with "iommu=none".
>>>
>>> Once the pluggable smmuv3 module is ready to test, we will make some
>>> change to libvirt for that and drop the auto-assigning patches from
>>> the VIRT code, so as to converge for a libvirt+QEMU test.
>>>
>>> FWIW, Robin requested a different solution for MSI mapping [1], v.s.
>>> the RMR one that we have been using since Eric's work. I drafted a
>>> few VFIO/IOMMUFD patches for that, yet paused for getting the vIOMMU
>>> series merged to this kernel cycle. I plan to continue in Nov/Dec.
>>> So, for the near term we will continue with the RMR solution, until
>>> we have something solid later.
>>>
>>> [1] https://lore.kernel.org/linux-iommu/ZrVN05VylFq8lK4q@Asurada-Nvidia/
>> At Red Hat we may find some cycles to resume working on the QEMU
>> integration. Please can you sketch some tasks we could carry out in
>> coordination with you and Shameer? Adding Don in the loop.
> That is great!
>
> So, given that Shameer is working on pluggable module part and we
> have folks working on libvirt. I think the only big thing here is
> the SMMUv3 series itself. Please refer to the changes in the link:
>  - cover-letter: Add HW accelerated nesting support for arm SMMUv3
>  - https://github.com/nicolinc/qemu/commits/wip/for_smmuv3_nesting-v4/
Looking at your branch I see the following series (marked with cover-letter)
*

  *

    cover-letter: Add RMR WAR for MSI mappings (based on former RMR flat
    mapping and not related to *[PATCH RFCv1 0/7] vfio: Allow userspace
    to specify the address for each MSI vector
    <https://lore.kernel.org/kvm/cover.1731130093.git.nicolinc@nvidia.com/#r>
    I guess)*

  *

    cover-letter: hw/arm/virt: Add multiple nested SMMUs (Nicolin ->
    Shameer)

  *

    cover-letter: Add HW accelerated nesting support for arm SMMUv3
    (Nicolin)

  *

    cover-letter: Add VIOMMU infrastructure support (Nicolin)

  *

    cover-letter: intel_iommu: Enable stage-1 translation for
    passthrough device (Zhenzhong)

  *

    cover-letter: intel_iommu: Enable stage-1 translation for emulated
    device (Zhenzhong)

The last one is covered by *[PATCH v5 00/20] intel_iommu: Enable stage-1
translation for emulated device
<20241111083457.2090664-1-zhenzhong.duan@intel.com/#r">https://lore.kernel.org/all/20241111083457.2090664-1-zhenzhong.duan@intel.com/#r>
*

*I see there is a reference to *"Enable stage-1 translation for
passthrough device" series but has it been posted for review? Adding
Zhenzhong in copy.

**

*Are there any posts upstream for the rest, besides Shameer's respin?
*

*
>
> I expect the IOMMU_HWPT_ALLOC (backend APIs) will go with Intel's
> series once their current "emulated devices" one gets merged. And
> I think I can prepare IOMMU_VIOMMU_ALLOC patches for backend APIs
> aligning with HWPT's.
Can you point us to the actual series including this IOMMU_HWPT_ALLOC
support? This would clarify which part you are going to work on next.

>
> That being said, one thing I am not sure is how we should bridge
> between these backend APIs and a virtual IOMMUs (vSMMU/intel). I
> think it'd be better for you and Red Hat to provide some insight,
> if calling the backend APIs directly from a viommu module isn't a
> preferable way.
can you clarify what you call backend API in that context?
>
> We also need your comments on vSMMU module patches that are still
> roughly a draft requiring a rework at some small details I think.
> So, if your (and Don's) bandwidth allows, perhaps you could take
> over the vSMMU patches? Otherwise, we can also share the task for
> reworking.
So we have started the multi smmu instantiation review and provided
feedbacks.
Which part can we work on then?

Thanks

Eric
>
> Thanks!
> Nicolin
>