[PATCH v1 00/15] virtio-mem: Expose device memory through multiple memslots

[David Hildenbrand]

This is the follow-up on [1] that took longer than expected because
vhost-user and other careless memslot consumers are just a pain. It's
essentially a complete rewrite now that we have
VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE (and I understood some nasty memslot
details better), so no changelog.

It's based on some cleanups [2, 3, 4] that are on the list.

Quoting from patch #13:

    Having large virtio-mem devices that only expose little memory to a VM
    is currently a problem: we map the whole sparse memory region into the
    guest using a single memslot, resulting in one gigantic memslot in KVM.
    KVM allocates metadata for the whole memslot, which can result in quite
    some memory waste.
    
    Assuming we have a 1 TiB virtio-mem device and only expose little (e.g.,
    1 GiB) memory, we would create a single 1 TiB memslot and KVM has to
    allocate metadata for that 1 TiB memslot: on x86, this implies allocating
    a significant amount of memory for metadata:
    
    (1) RMAP: 8 bytes per 4 KiB, 8 bytes per 2 MiB, 8 bytes per 1 GiB
        -> For 1 TiB: 2147483648 + 4194304 + 8192 = ~ 2 GiB (0.2 %)
    
        With the TDP MMU (cat /sys/module/kvm/parameters/tdp_mmu) this gets
        allocated lazily when required for nested VMs
    (2) gfn_track: 2 bytes per 4 KiB
        -> For 1 TiB: 536870912 = ~512 MiB (0.05 %)
    (3) lpage_info: 4 bytes per 2 MiB, 4 bytes per 1 GiB
        -> For 1 TiB: 2097152 + 4096 = ~2 MiB (0.0002 %)
    (4) 2x dirty bitmaps for tracking: 2x 1 bit per 4 KiB page
        -> For 1 TiB: 536870912 = 64 MiB (0.006 %)
    
    So we primarily care about (1) and (2). The bad thing is, that the
    memory consumption *doubles* once SMM is enabled, because we create the
    memslot once for !SMM and once for SMM.
    
    Having a 1 TiB memslot without the TDP MMU consumes around:
    * With SMM: 5 GiB
    * Without SMM: 2.5 GiB
    Having a 1 TiB memslot with the TDP MMU consumes around:
    * With SMM: 1 GiB
    * Without SMM: 512 MiB
    
    ... and that's really something we want to optimize, to be able to just
    start a VM with small boot memory (e.g., 4 GiB) and a virtio-mem device
    that can grow very large (e.g., 1 TiB).
    
    Consequently, using multiple memslots and only mapping the memslots we
    really need can significantly reduce memory waste and speed up
    memslot-related operations. Let's expose the sparse RAM memory region using
    multiple memslots, mapping only the memslots we currently need into our
    device memory region container.

The feature has to be turned manually on ("multiple-memslots=on") for now.
As virtio-mem will auto-detect the number of memslots to use based on the
current memslot configuration (max 256 for all memory devices), it's
suggested to:

(1) Define vhost device before memory devices on the QEMU cmdline. This
    primarily applies to vhost-user devices.
(2) Not hotplug vhost (especially vhost-user) devices that impose further
    memslot retsrictions. For example, hotplugging a vhost-user device
    might now fail because more memslots are getting ised.

Once vhost-user supports more memslots (i.e., ~512 instead of 32) these
restrictions no longer apply.

I run some tests on x86_64, but have to do some more testing with vfio.


Patch #1-#4 add a soft limit for memslots, based on the maximum number of
memslots around.

Patch #5-#7 add support for memory devices that statically consome multiple
memslots.

Patch #8-#9 add support for memory devices that dynamically consume
memslots on demand.

Patch #10-#11 adds support for memory devices that can auto-detect the
number of memslots to use.

Patch #12-#13 adds "multiple-memslots=on" support to virtio-mem

Patch #14-#15 make sure that virtio-mem memslots can be enabled/disable
atomically

[1] https://lkml.kernel.org/r/20211013103330.26869-1-david@redhat.com
[2] https://lkml.kernel.org/r/20230523185915.540373-1-david@redhat.com
[3] https://lkml.kernel.org/r/20230530113838.257755-1-david@redhat.com
[4] https://lkml.kernel.org/r/20230523183036.517957-1-david@redhat.com

Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Xiao Guangrong <xiaoguangrong.eric@gmail.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: "Philippe Mathieu-Daudé" <philmd@linaro.org>
Cc: Eduardo Habkost <eduardo@habkost.net>
Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
Cc: Yanan Wang <wangyanan55@huawei.com>
Cc: Michal Privoznik <mprivozn@redhat.com>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Gavin Shan <gshan@redhat.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Cc: kvm@vger.kernel.org

David Hildenbrand (15):
  memory-device: Track the required memslots in DeviceMemoryState
  kvm: Add stub for kvm_get_max_memslots()
  vhost: Add vhost_get_max_memslots()
  memory-device,vhost: Add a memslot soft limit for memory devices
  kvm: Return number of free memslots
  vhost: Return number of free memslots
  memory-device: Support memory devices that statically consume multiple
    memslots
  memory-device: Track the actually used memslots in DeviceMemoryState
  memory-device,vhost: Support memory devices that dynamically consume
    multiple memslots
  pc-dimm: Provide pc_dimm_get_free_slots() to query free ram slots
  memory-device: Support memory-devices with auto-detection of the
    number of memslots
  memory: Clarify mapping requirements for RamDiscardManager
  virtio-mem: Expose device memory via multiple memslots if enabled
  memory,vhost: Allow for marking memory device memory regions
    unmergeable
  virtio-mem: Mark memslot alias memory regions unmergeable

 accel/kvm/kvm-all.c            |  35 +++--
 accel/stubs/kvm-stub.c         |   9 +-
 hw/mem/memory-device.c         | 263 ++++++++++++++++++++++++++++++--
 hw/mem/pc-dimm.c               |  27 ++++
 hw/virtio/vhost-stub.c         |   9 +-
 hw/virtio/vhost.c              |  41 ++++-
 hw/virtio/virtio-mem-pci.c     |  21 +++
 hw/virtio/virtio-mem.c         | 271 ++++++++++++++++++++++++++++++++-
 include/exec/memory.h          |  27 +++-
 include/hw/boards.h            |  10 +-
 include/hw/mem/memory-device.h |  37 +++++
 include/hw/mem/pc-dimm.h       |   1 +
 include/hw/virtio/vhost.h      |   3 +-
 include/hw/virtio/virtio-mem.h |  23 ++-
 include/sysemu/kvm.h           |   4 +-
 include/sysemu/kvm_int.h       |   1 +
 softmmu/memory.c               |  35 ++++-
 stubs/qmp_memory_device.c      |  15 ++
 18 files changed, 777 insertions(+), 55 deletions(-)

-- 
2.40.1