Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3

[Nathan Chen]

>  >> Also as a heads up, I've added support for auto-inserting PCIe switch
>  >> between the PXB and GPUs in libvirt to attach multiple devices to a
> SMMU
>  >> node per libvirt's documentation - "If you intend to plug multiple
>  >> devices into a pcie-expander-bus, you must connect a
>  >> pcie-switch-upstream-port to the pcie-root-port that is plugged 
into the
>  >> pcie-expander-bus, and multiple pcie-switch-downstream-ports to the
>  >> pcie-switch-upstream-port". Future unit-tests should follow this
>  >> topology configuration.
>  >
> >  > Ok. Could you please give me an example Qemu equivalent command
> > option,
> >  > if possible, for the above case. I am not that familiar with libvirt
> > and I would
> >  > also like to test the above scenario.
> >
> > You can use "-device x3130-upstream" for the upstream switch port, and
> > "-device xio3130-downstream" for the downstream port:
> >
> >   -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x1 \
> >   -device pcie-root-port,id=pci.2,bus=pci.1,addr=0x0 \
> >   -device x3130-upstream,id=pci.3,bus=pci.2,addr=0x0 \
> >   -device xio3130-
> > downstream,id=pci.4,bus=pci.3,addr=0x0,chassis=17,port=1 \
> >   -device vfio-pci,host=0009:01:00.0,id=hostdev0,bus=pci.4,addr=0x0 \
> >   -device arm-smmuv3-nested,pci-bus=pci.1
>
> Thanks. Just wondering why libvirt mandates usage of pcie-switch for 
multiple
> device plugging rather than just using pcie-root-ports?
>
> Please let me if there is any advantage in doing so that you are 
aware > of.

Actually it seems like that documentation I quoted is out of date. That 
section of the documentation for pcie-expander-bus was written before a 
patch that revised libvirt's pxb to have 32 slots instead of just 1 
slot, and it wasn't updated afterwards.

With your branch and my libvirt prototype, I was still able to attach a 
passthrough device behind a PCIe switch and see it attached to a vSMMU 
in the VM, so I'm not sure if you need to make additional changes to 
your solution to support this. But I think we should still support/test 
the case where VFIO devices are behind a switch, otherwise we're placing 
a limitation on end users who have a use case for it.

-Nathan