Re: [Qemu-devel] [RFC PATCH] pci: Use PCI aliases when determining device IOMMU address space

[Singh, Brijesh]

On 7/24/19 2:42 PM, Alex Williamson wrote:
> On Wed, 24 Jul 2019 08:43:55 -0600
> Alex Williamson <address@hidden> wrote:
> 
>> On Wed, 24 Jul 2019 18:03:31 +0800
>> Peter Xu <address@hidden> wrote:
>>
>>> On Wed, Jul 24, 2019 at 05:39:22AM -0400, Michael S. Tsirkin wrote:
>>>> On Wed, Jul 24, 2019 at 03:14:39PM +0800, Peter Xu wrote:
>>>>> On Tue, Jul 23, 2019 at 11:26:18AM -0600, Alex Williamson wrote:
>>>>>>> On 3/29/19 11:49 AM, Alex Williamson wrote:
>>>>>>>> [Cc +Brijesh]
>>>>>>>>
>>>>>>>> Hi Brijesh, will the change below require the IVRS to be updated to
>>>>>>>> include aliases for all BDF ranges behind a conventional bridge?  I
>>>>>>>> think the Linux code handles this regardless of the firmware provided
>>>>>>>> aliases, but is it required per spec for the ACPI tables to include
>>>>>>>> bridge aliases?  Thanks,
> [snip]
> 
> For a data point, I fired up an old 990FX system which includes a
> PCIe-to-PCI bridge and I added a plugin PCIe-to-PCI bridge just to keep
> things interesting... guess how many alias ranges are in the IVRS...
> Yep, just the one built into the motherboard:
> 
> AMD-Vi: Using IVHD type 0x10
> AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
> AMD-Vi:        mmio-addr: 00000000fec30000
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 00:00.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 00:00.2
> AMD-Vi:   DEV_SELECT                   devid: 00:09.0 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 01:00.0 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 00:0a.0 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 02:00.0 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 00:11.0 flags: 00
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 00:12.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 00:12.2
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 00:13.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 00:13.2
> AMD-Vi:   DEV_SELECT                   devid: 00:14.0 flags: d7
> AMD-Vi:   DEV_SELECT                   devid: 00:14.2 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 00:14.3 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 00:14.4 flags: 00
> AMD-Vi:   DEV_ALIAS_RANGE              devid: 03:00.0 flags: 00 devid_to: 
> 00:14.4
> AMD-Vi:   DEV_RANGE_END                devid: 03:1f.7
> 
> [Everything on bus 03:xx.x is aliased to device 00:14.4, the builtin 
> PCIe-to-PCI bridge]
> 
> AMD-Vi:   DEV_SELECT                   devid: 00:14.5 flags: 00
> AMD-Vi:   DEV_SELECT                   devid: 00:15.0 flags: 00
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 04:00.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 04:1f.7
> AMD-Vi:   DEV_SELECT                   devid: 00:15.1 flags: 00
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 05:00.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 05:1f.7
> AMD-Vi:   DEV_SELECT                   devid: 00:15.2 flags: 00
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 06:00.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 06:1f.7
> AMD-Vi:   DEV_SELECT                   devid: 00:15.3 flags: 00
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 08:00.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 08:1f.7
> AMD-Vi:   DEV_SELECT_RANGE_START       devid: 00:16.0 flags: 00
> AMD-Vi:   DEV_RANGE_END                devid: 00:16.2
> AMD-Vi:   DEV_SPECIAL(IOAPIC[8])              devid: 00:14.0
> AMD-Vi:   DEV_SPECIAL(HPET[0])                devid: 00:14.0
> AMD-Vi:   DEV_SPECIAL(IOAPIC[8])              devid: 00:00.1
> 
> -[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host 
> Bridge
>             +-00.2  Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O 
> Memory Management Unit (IOMMU)
>             +-09.0-[01]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host 
> Controller
>             +-0a.0-[02]----00.0  Marvell Technology Group Ltd. 88SE9172 SATA 
> 6Gb/s Controller
>             +-11.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> SATA Controller [AHCI mode]
>             +-12.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB OHCI0 Controller
>             +-12.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB EHCI Controller
>             +-13.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB OHCI0 Controller
>             +-13.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB EHCI Controller
>             +-14.0  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus 
> Controller
>             +-14.2  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia 
> (Intel HDA)
>             +-14.3  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> LPC host controller
>             +-14.4-[03]--+-06.0  NVidia / SGS Thomson (Joint Venture) Riva128
>             |            \-0e.0  VIA Technologies, Inc. VT6306/7/8 [Fire 
> II(M)] IEEE 1394 OHCI Controller
>             +-14.5  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB OHCI2 Controller
>             +-15.0-[04]----00.0  Realtek Semiconductor Co., Ltd. 
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>             +-15.1-[05]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host 
> Controller
>             +-15.2-[06-07]----00.0-[07]----00.0  Realtek Semiconductor Co., 
> Ltd. Device 8149
>             +-15.3-[08]--
>             +-16.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB OHCI0 Controller
>             +-16.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
> USB EHCI Controller
>             +-18.0  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
> HyperTransport Configuration
>             +-18.1  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
> Address Map
>             +-18.2  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
> DRAM Controller
>             +-18.3  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
> Miscellaneous Control
>             \-18.4  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
> Link Control
> 
> 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI 
> Bridge (rev 40) (prog-if 01 [Subtractive decode])
>       Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
> 
> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge 
> (rev 03) (prog-if 00 [Normal decode])
>       Bus: primary=06, secondary=07, subordinate=07, sec-latency=32
>       Capabilities: [80] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
> 
> I realized as I was writing, that the alias caused by 06:00.0 would be
> devfn 00.0 on the secondary bus 07, ie. 07:00.0 would alias to
> 07:00.0, so technically there's really not an alias for this small
> case.  So I replaced the NIC with this:
> 
>             +-15.2-[06-07]----00.0-[07]--+-00.0  NEC Corporation OHCI USB 
> Controller
>                                          +-00.1  NEC Corporation OHCI USB 
> Controller
>                                          \-00.2  NEC Corporation uPD72010x 
> USB 2.0 Controller
> 
> Now functions 07:00.[12] also alias to 07:00.0.  The IVRS table is
> unaffected.
> 
> I'm tempted to say that QEMU would be no worse than bare metal if we
> simply ignore IVHD device alias entries.  I know that Linux will figure
> out the aliasing regardless of the IVRS.  Will Windows guests?  I'd
> love to hear from Bijesh or Suravee regarding the behavior above versus
> what AMD expects from system vendors.  Thanks,
> 

Alex,

I am not well versed on the expected IOMMU address space behavior yet,
Suravee will know more about. I will ask him to take a look at this
thread.

thanks

> Alex
>