Re: [PATCH v6 5/5] hw/pci: ensure PCIE devices are plugged into only slot 0 of PCIE port

[Michael S. Tsirkin]

On Fri, Jun 30, 2023 at 11:43:15AM +0900, Akihiko Odaki wrote:
> On 2023/06/29 23:18, Ani Sinha wrote:
> > 
> > 
> > > On 29-Jun-2023, at 2:19 PM, Akihiko Odaki <akihiko.odaki@daynix.com> 
> > > wrote:
> > > 
> > > On 2023/06/29 17:05, Ani Sinha wrote:
> > > > On Thu, 29 Jun, 2023, 12:17 pm Akihiko Odaki, <akihiko.odaki@daynix.com 
> > > > <mailto:akihiko.odaki@daynix.com>> wrote:
> > > >     On 2023/06/29 13:07, Ani Sinha wrote:
> > > >      > PCI Express ports only have one slot, so PCI Express devices can
> > > >     only be
> > > >      > plugged into slot 0 on a PCIE port. Enforce it.
> > > >      >
> > > >      > The change has been tested to not break ARI by instantiating
> > > >     seven vfs on an
> > > >      > emulated igb device (the maximum number of vfs the linux igb
> > > >     driver supports).
> > > >      > The vfs are seen to have non-zero device/slot numbers in the
> > > >     conventional
> > > >      > PCI BDF representation.
> > > >      >
> > > >      > CC: jusual@redhat.com <mailto:jusual@redhat.com>
> > > >      > CC: imammedo@redhat.com <mailto:imammedo@redhat.com>
> > > >      > CC: akihiko.odaki@daynix.com <mailto:akihiko.odaki@daynix.com>
> > > >      >
> > > >      > Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2128929
> > > >     <https://bugzilla.redhat.com/show_bug.cgi?id=2128929>
> > > >      > Signed-off-by: Ani Sinha <anisinha@redhat.com
> > > >     <mailto:anisinha@redhat.com>>
> > > >      > Reviewed-by: Julia Suvorova <jusual@redhat.com
> > > >     <mailto:jusual@redhat.com>>
> > > >      > ---
> > > >      >   hw/pci/pci.c | 15 +++++++++++++++
> > > >      >   1 file changed, 15 insertions(+)
> > > >      >
> > > >      > diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> > > >      > index e2eb4c3b4a..0320ac2bb3 100644
> > > >      > --- a/hw/pci/pci.c
> > > >      > +++ b/hw/pci/pci.c
> > > >      > @@ -65,6 +65,7 @@ bool pci_available = true;
> > > >      >   static char *pcibus_get_dev_path(DeviceState *dev);
> > > >      >   static char *pcibus_get_fw_dev_path(DeviceState *dev);
> > > >      >   static void pcibus_reset(BusState *qbus);
> > > >      > +static bool pcie_has_upstream_port(PCIDevice *dev);
> > > >      >
> > > >      >   static Property pci_props[] = {
> > > >      >       DEFINE_PROP_PCI_DEVFN("addr", PCIDevice, devfn, -1),
> > > >      > @@ -1190,6 +1191,20 @@ static PCIDevice
> > > >     *do_pci_register_device(PCIDevice *pci_dev,
> > > >      >                      name);
> > > >      >
> > > >      >          return NULL;
> > > >      > +    } /*
> > > >      > +       * With SRIOV and ARI, vfs can have non-zero slot in the
> > > >     conventional
> > > >      > +       * PCI interpretation as all five bits reserved for slot
> > > >     addresses are
> > > >      > +       * also used for function bits for the various vfs. Ignore
> > > >     that case.
> > > >      > +       * It is too early here to check for ARI capabilities in
> > > >     the PCI config
> > > >      > +       * space. Hence, we check for a vf device instead.
> > > >      > +       */
> > > >     Why don't just perform this check after the capabilities are set?
> > > > We don't want to allocate resources for wrong device parameters. We 
> > > > want to error out early. Other checks also are performed at the same 
> > > > place .
> > > 
> > > It is indeed better to raise an error as early as possible so that we can 
> > > avoid allocation and other operations that will be reverted and may go 
> > > wrong due to the invalid condition. That should be the reason why other 
> > > checks for the address are performed here.
> > > 
> > > However, in this particular case, we cannot confidently perform the check 
> > > here because it is unknown if the ARI capability will be advertised until 
> > > the device realization code runs. This can justify delaying the check 
> > > after the device realization, unlike the other checks.
> > 
> > Ok so are you proposing that the check we have right before (the check for 
> > unoccupied function 0) be also moved? It also uses the same vf 
> > approximation for seemingly to support ARI.
> 
> No, I don't think the check for function 0 is required to be disabled
> because of the change of addressing caused by ARI, but it is required
> because SR-IOV VF can be added and removed while the PF (function 0)
> remains. I think this check should be performed also when SR-IOV is disabled
> and ARI is enabled.
> 
> Thus the check for unoccupied function 0 needs to use pci_is_vf() instead of
> checking ARI capability, and that can happen in do_pci_register_device().
> 
> > Also where do you propose we move the check?
> 
> In pci_qdev_realize(), somewhere after pc->realize() and before option ROM
> loading. See the check for failover pair as an example. I guess it's also
> placed in this region because it needs capability information.

How about instead of spending so much time on working around
incomplete ARI support we actually complete ARI support?



> > 
> > > 
> > > > Show quoted text
> > > >     Regards,
> > > >     Akihiko Odaki
> > > >      > +    else if (!pci_is_vf(pci_dev) &&
> > > >      > +             pcie_has_upstream_port(pci_dev) &&
> > > >      > +             PCI_SLOT(devfn)) {
> > > >      > +        error_setg(errp, "PCI: slot %d is not valid for %s,"
> > > >      > +                   " parent device only allows plugging into
> > > >     slot 0.",
> > > >      > +                   PCI_SLOT(devfn), name);
> > > >      > +        return NULL;
> > > >      >       }
> > > >      >
> > > >      >       pci_dev->devfn = devfn;
> >