Re: XIVE VFIO kernel resample failure in INTx mode under heavy load

[Timothy Pearson]

----- Original Message -----
> From: "Cédric Le Goater" <clg@kaod.org>
> To: "Alexey Kardashevskiy" <aik@ozlabs.ru>, "Alex Williamson" 
> <alex.williamson@redhat.com>, "Timothy Pearson"
> <tpearson@raptorengineering.com>
> Cc: "list@suse.de:PowerPC" <qemu-ppc@nongnu.org>, "qemu-devel" 
> <qemu-devel@nongnu.org>, "Frederic Barrat"
> <fbarrat@linux.ibm.com>, "npiggin" <npiggin@gmail.com>, "David Gibson" 
> <david@gibson.dropbear.id.au>
> Sent: Thursday, April 21, 2022 1:35:50 AM
> Subject: Re: XIVE VFIO kernel resample failure in INTx mode under heavy load

> On 4/21/22 05:07, Alexey Kardashevskiy wrote:
>> 
>> 
>> On 14/04/2022 22:41, Cédric Le Goater wrote:
>>>
>>>>> After re-reading what I just wrote, I am leaning towards disabling use of
>>>>> KVM_CAP_IRQFD_RESAMPLE as it seems last worked on POWER8 and never since 
>>>>> :)
>>>>>
>>>>> Did I miss something in the picture (hey Cedric)?
>>>>
>>>> How about disabling it like this?
>>>>
>>>> =====
>>>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>>>> index 5bfd4aa9e5aa..c999f7b1ab1b 100644
>>>> --- a/hw/ppc/spapr_pci.c
>>>> +++ b/hw/ppc/spapr_pci.c
>>>> @@ -732,7 +732,7 @@ static PCIINTxRoute spapr_route_intx_pin_to_irq(void
>>>> *opaque, int pin)
>>>>       SpaprPhbState *sphb = SPAPR_PCI_HOST_BRIDGE(opaque);
>>>>       PCIINTxRoute route;
>>>>
>>>> -    route.mode = PCI_INTX_ENABLED;
>>>> +    route.mode = PCI_INTX_DISABLED;
>>>>
>>>> =====
>>>
>>> I like it.
>> 
>> 
>> The only thing is that this resampling works on POWER8/XICS and
>> removing it there is not great. So far sPAPR PHB was unaware of
>> underlying interrupt controller, or was not it?
> 
> It is. The dynamic change of the underlying irqchip in QEMU and
> in KVM required that for CAS. Of course, plenty is done in the
> back of the devices when this happens, see spapr_irq.
> 
> There are some quirks related to LPM with VIO devices in Linux.
> This is the only case I know about.
> 
> Thanks,
> 
> C.

Unfortunately this remains quite broken, and after a kernel upgrade (including 
the purported fix [1]) and a qemu upgrade we have now completely lost the 
ability to get the card working in the guest with *any* combination of 
parameters.

In guest XIVE mode with irqchip on it passes through a handful of interrupts, 
then dies.  In guest XICS mode we're dropping the majority of the interrupts.  
This is all on POWER9.

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/powerpc/kvm/powerpc.c?id=52882b9c7a761b2b4e44717d6fbd1ed94c601b7f