Re: [Qemu-devel] [RFC] map 64-bit PCI devices after all possible RAM

[Gerd Hoffmann]

  Hi,

> So far from QEMU side it's partially (only memory region mapping and not ACPI
> window) configurable via {i440FX-pcihost|q35-pcihost}.pci-hole64-size property

/me looks.

Hmm, so the pci-hole64 memory region basically covers all non-memory
area, leaving no free space.

> > The window location can either be made configurable too, or we simply
> > place it at the top of the address space, with "address space" being
> > what the cpu can address according to cpuinfo.
> An earlier attempt by Michael to push complete PCI window placement info
> via "etc/pci-info" romfile to Seabios was rejected in favor of letting Seabios
> to program windows at hardcoded(32-bit/behind high mem) locations with a
> 64-bit window size (in ACPI) that covers all present devices but doesn't
> account for future PCI hotplug either.

Correct.  The ACPI tables should reflect what SeaBIOS has programmed, to
avoid nasty dependencies between seabios and qemu.

The same should apply to pci-hole64 IMO.

> That behavior maintained in his "ACPI in QEMU" series, see:
> http://patchwork.ozlabs.org/patch/281032/
> acpi_get_pci_info()->i440fx_pcihost_get_pci_hole64_end()->pci_bus_get_w64_range()
> which is then embedded in ACPI table. So end result stays the same as
> before (no usable 64-bit PCI window for hotlug).

Yes.  And if we change seabios to do something else qemu nicely adapts
to that, without requiring us to update things in lockstep.

> But 64-bit PCI window size, which is capped by QEMU to insane legacy 62 bits
> (memory region size), is a bit of orthogonal to freeing space for memory
> hotplug before it.

Yep.  So seabios should leave some free address space for memory
hotplug.  And if we change seabios to map the 64bit pci bars somewhere
else we should also allow for a larger 64bit pci window to get some
address space for pci hotplug.

If we can do that without hints from the qemu I'd prefer that.

> > 40 address lines allow 1TB, so we would place the window just below 1TB.
> > 
> > Comments?
> More to the point if OS supports/enforces 1Tb physical address space,the RAM
> and 64-bit PCI hole are going to contend for it, QEMU could abort on startup
> if they both do not fit in CPU supported address space but I don't see what
> else it could do.

Yes.

> Proposed patch favors RAM vs 64-bit PCI hole and moves the hole behind the
> possible RAM, which in present state of QEMU potentially leaves the rest of
> address space up to 62 bits for hole.

So you'd end up with the 64bit hole being above the address space the
virtual cpu claims to support.  Not exactly nice either.  Maybe things
work nevertheless, maybe not ...

Both cases can easily be fixed by just using a cpu with enough physical
address lines to fit everything in, so I don't think we should bother
too much about this corner case.

Just in case this wasn't clear: my idea is that seabios figures the
address space size at runtime, so the 1TB would NOT be hard-coded, it
just served as example with the current default qemu cpu.

So with my idea the address space would have all RAM at the bottom
(well, starting at 4g).  All PCI devices at the top.  Free space for
hotplug inbetween.  RAM can grow up.  PCI space can grow down.

Note that qemu can make 64bit pci window in the acpi tables larger than
what is actually used by the mapped bars, to make room for hotplugging,
without any help from seabios (once the acpi table generation patches
are merged).  So with the current seabios (bars mapped above memory) it
can set the end address higher.  When seabios starts mapping the pci
bars high it can set the start address lower.

Anyone has a use case not handled by this approach?

> It has drawback that one can't get a working VM if QEMU is started in
> memory hotlug mode with old BIOS + PCI devices that require 64-bit bars,
> otherwise it's backward compatible.

Yes.  Updating seabios will be needed to use memory hotplug together
with 64bit pci no matter how we tackle the issue.