Re: [Qemu-devel] [RFC PATCH] pci: Use PCI aliases when determining device IOMMU address space

[Alex Williamson]

On Wed, 24 Jul 2019 08:43:55 -0600
Alex Williamson <address@hidden> wrote:

> On Wed, 24 Jul 2019 18:03:31 +0800
> Peter Xu <address@hidden> wrote:
> 
> > On Wed, Jul 24, 2019 at 05:39:22AM -0400, Michael S. Tsirkin wrote:  
> > > On Wed, Jul 24, 2019 at 03:14:39PM +0800, Peter Xu wrote:    
> > > > On Tue, Jul 23, 2019 at 11:26:18AM -0600, Alex Williamson wrote:    
> > > > > > On 3/29/19 11:49 AM, Alex Williamson wrote:    
> > > > > > > [Cc +Brijesh]
> > > > > > > 
> > > > > > > Hi Brijesh, will the change below require the IVRS to be updated 
> > > > > > > to
> > > > > > > include aliases for all BDF ranges behind a conventional bridge?  
> > > > > > > I
> > > > > > > think the Linux code handles this regardless of the firmware 
> > > > > > > provided
> > > > > > > aliases, but is it required per spec for the ACPI tables to 
> > > > > > > include
> > > > > > > bridge aliases?  Thanks,
[snip]

For a data point, I fired up an old 990FX system which includes a
PCIe-to-PCI bridge and I added a plugin PCIe-to-PCI bridge just to keep
things interesting... guess how many alias ranges are in the IVRS...
Yep, just the one built into the motherboard:

AMD-Vi: Using IVHD type 0x10
AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
AMD-Vi:        mmio-addr: 00000000fec30000
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 00:00.2
AMD-Vi:   DEV_SELECT                     devid: 00:09.0 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 01:00.0 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 00:0a.0 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 02:00.0 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 00:11.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:12.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 00:12.2
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:13.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 00:13.2
AMD-Vi:   DEV_SELECT                     devid: 00:14.0 flags: d7
AMD-Vi:   DEV_SELECT                     devid: 00:14.2 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 00:14.3 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 00:14.4 flags: 00
AMD-Vi:   DEV_ALIAS_RANGE                devid: 03:00.0 flags: 00 devid_to: 
00:14.4
AMD-Vi:   DEV_RANGE_END          devid: 03:1f.7

[Everything on bus 03:xx.x is aliased to device 00:14.4, the builtin 
PCIe-to-PCI bridge]

AMD-Vi:   DEV_SELECT                     devid: 00:14.5 flags: 00
AMD-Vi:   DEV_SELECT                     devid: 00:15.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 04:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 04:1f.7
AMD-Vi:   DEV_SELECT                     devid: 00:15.1 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 05:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 05:1f.7
AMD-Vi:   DEV_SELECT                     devid: 00:15.2 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 06:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 06:1f.7
AMD-Vi:   DEV_SELECT                     devid: 00:15.3 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 08:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 08:1f.7
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:16.0 flags: 00
AMD-Vi:   DEV_RANGE_END          devid: 00:16.2
AMD-Vi:   DEV_SPECIAL(IOAPIC[8])                devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(HPET[0])          devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(IOAPIC[8])                devid: 00:00.1

-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host 
Bridge
           +-00.2  Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O 
Memory Management Unit (IOMMU)
           +-09.0-[01]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host 
Controller
           +-0a.0-[02]----00.0  Marvell Technology Group Ltd. 88SE9172 SATA 
6Gb/s Controller
           +-11.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 
SATA Controller [AHCI mode]
           +-12.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
OHCI0 Controller
           +-12.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
EHCI Controller
           +-13.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
OHCI0 Controller
           +-13.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
EHCI Controller
           +-14.0  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller
           +-14.2  Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel 
HDA)
           +-14.3  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC 
host controller
           +-14.4-[03]--+-06.0  NVidia / SGS Thomson (Joint Venture) Riva128
           |            \-0e.0  VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] 
IEEE 1394 OHCI Controller
           +-14.5  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
OHCI2 Controller
           +-15.0-[04]----00.0  Realtek Semiconductor Co., Ltd. 
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           +-15.1-[05]----00.0  Etron Technology, Inc. EJ168 USB 3.0 Host 
Controller
           +-15.2-[06-07]----00.0-[07]----00.0  Realtek Semiconductor Co., Ltd. 
Device 8149
           +-15.3-[08]--
           +-16.0  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
OHCI0 Controller
           +-16.2  Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB 
EHCI Controller
           +-18.0  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
HyperTransport Configuration
           +-18.1  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
Address Map
           +-18.2  Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM 
Controller
           +-18.3  Advanced Micro Devices, Inc. [AMD] Family 10h Processor 
Miscellaneous Control
           \-18.4  Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link 
Control

00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI 
Bridge (rev 40) (prog-if 01 [Subtractive decode])
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=64

06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge 
(rev 03) (prog-if 00 [Normal decode])
        Bus: primary=06, secondary=07, subordinate=07, sec-latency=32
        Capabilities: [80] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00

I realized as I was writing, that the alias caused by 06:00.0 would be
devfn 00.0 on the secondary bus 07, ie. 07:00.0 would alias to
07:00.0, so technically there's really not an alias for this small
case.  So I replaced the NIC with this:

           +-15.2-[06-07]----00.0-[07]--+-00.0  NEC Corporation OHCI USB 
Controller
                                        +-00.1  NEC Corporation OHCI USB 
Controller
                                        \-00.2  NEC Corporation uPD72010x USB 
2.0 Controller

Now functions 07:00.[12] also alias to 07:00.0.  The IVRS table is
unaffected.

I'm tempted to say that QEMU would be no worse than bare metal if we
simply ignore IVHD device alias entries.  I know that Linux will figure
out the aliasing regardless of the IVRS.  Will Windows guests?  I'd
love to hear from Bijesh or Suravee regarding the behavior above versus
what AMD expects from system vendors.  Thanks,

Alex