Re: [Qemu-devel] [RFC] map 64-bit PCI devices after all possible RAM

[Michael S. Tsirkin]

On Thu, Oct 10, 2013 at 12:56:23PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > So far from QEMU side it's partially (only memory region mapping and not 
> > ACPI
> > window) configurable via {i440FX-pcihost|q35-pcihost}.pci-hole64-size 
> > property
> 
> /me looks.
> 
> Hmm, so the pci-hole64 memory region basically covers all non-memory
> area, leaving no free space.

This is kind of derived from the PIIX spec although of course
it did not discuss 64 bit memory.

> > > The window location can either be made configurable too, or we simply
> > > place it at the top of the address space, with "address space" being
> > > what the cpu can address according to cpuinfo.
> > An earlier attempt by Michael to push complete PCI window placement info
> > via "etc/pci-info" romfile to Seabios was rejected in favor of letting 
> > Seabios
> > to program windows at hardcoded(32-bit/behind high mem) locations with a
> > 64-bit window size (in ACPI) that covers all present devices but doesn't
> > account for future PCI hotplug either.
> 
> Correct.  The ACPI tables should reflect what SeaBIOS has programmed, to
> avoid nasty dependencies between seabios and qemu.
> 
> The same should apply to pci-hole64 IMO.
> 
> > That behavior maintained in his "ACPI in QEMU" series, see:
> > http://patchwork.ozlabs.org/patch/281032/
> > acpi_get_pci_info()->i440fx_pcihost_get_pci_hole64_end()->pci_bus_get_w64_range()
> > which is then embedded in ACPI table. So end result stays the same as
> > before (no usable 64-bit PCI window for hotlug).
> 
> Yes.  And if we change seabios to do something else qemu nicely adapts
> to that, without requiring us to update things in lockstep.
> 
> > But 64-bit PCI window size, which is capped by QEMU to insane legacy 62 bits
> > (memory region size), is a bit of orthogonal to freeing space for memory
> > hotplug before it.
> 
> Yep.  So seabios should leave some free address space for memory
> hotplug.  And if we change seabios to map the 64bit pci bars somewhere
> else we should also allow for a larger 64bit pci window to get some
> address space for pci hotplug.
> 
> If we can do that without hints from the qemu I'd prefer that.

I think the simplest way to do all this is simply to tell seabios
that we have more memory. seabios already programs 64 bit BARs
higher than memory.

No new interface seems necessary.


> > > 40 address lines allow 1TB, so we would place the window just below 1TB.
> > > 
> > > Comments?
> > More to the point if OS supports/enforces 1Tb physical address space,the RAM
> > and 64-bit PCI hole are going to contend for it, QEMU could abort on startup
> > if they both do not fit in CPU supported address space but I don't see what
> > else it could do.
> 
> Yes.
> 
> > Proposed patch favors RAM vs 64-bit PCI hole and moves the hole behind the
> > possible RAM, which in present state of QEMU potentially leaves the rest of
> > address space up to 62 bits for hole.
> 
> So you'd end up with the 64bit hole being above the address space the
> virtual cpu claims to support.  Not exactly nice either.  Maybe things
> work nevertheless, maybe not ...
> 
> Both cases can easily be fixed by just using a cpu with enough physical
> address lines to fit everything in, so I don't think we should bother
> too much about this corner case.
> 
> Just in case this wasn't clear: my idea is that seabios figures the
> address space size at runtime, so the 1TB would NOT be hard-coded, it
> just served as example with the current default qemu cpu.
> 
> So with my idea the address space would have all RAM at the bottom
> (well, starting at 4g).  All PCI devices at the top.  Free space for
> hotplug inbetween.  RAM can grow up.  PCI space can grow down.
> 
> Note that qemu can make 64bit pci window in the acpi tables larger than
> what is actually used by the mapped bars, to make room for hotplugging,
> without any help from seabios (once the acpi table generation patches
> are merged).  So with the current seabios (bars mapped above memory) it
> can set the end address higher.  When seabios starts mapping the pci
> bars high it can set the start address lower.
> 
> Anyone has a use case not handled by this approach?

I think the issue is with legacy guests.
E.g. if VCPU claims to support 50 bit of memory
do we put high PCI memory at 1 << 50?
If yes old guests which expect at most 40 bit
will not be able to use it.


> > It has drawback that one can't get a working VM if QEMU is started in
> > memory hotlug mode with old BIOS + PCI devices that require 64-bit bars,
> > otherwise it's backward compatible.
> 
> Yes.  Updating seabios will be needed to use memory hotplug together
> with 64bit pci no matter how we tackle the issue.
>