virito-mem currently relies on having a single sparse memory region (anon
mmap, mmaped file, mmaped huge pages, mmap shmem) per VM. Although we can
share memory with other processes, sharing with other VMs is not intended.
Instead of actually mmaping parts dynamically (which can be quite
expensive), virtio-mem relies on punching holes into the backend and
dynamically allocating memory/file blocks/... on access.
So the easy way to make it work is:
a) Exposing the CXL memory to the buddy via dax/kmem, esulting in device
memory getting managed by the buddy on a separate NUMA node.
Linux kernel buddy system? how to guarantee other applications don't apply
memory
from it
Excellent question. Usually, you would online the memory to ZONE_MOVABLE,
such that even if some other allocation ended up there, that it could
get migrated somewhere else.
For example, "daxctl reconfigure-device" tries doing that as default:
https://pmem.io/ndctl/daxctl-reconfigure-device.html
However, I agree that we might actually want to tell the system to not
use this CPU-less node as fallback for other allocations, and that we
might not want to swap out such memory etc.
But, in the end all that virtio-mem needs to work in the hypervisor is
a) A sparse memmap (anonymous RAM, memfd, file)
b) A way to populate memory within that sparse memmap (e.g., on fault,
using madvise(MADV_POPULATE_WRITE), fallocate())
c) A way to discard memory (madvise(MADV_DONTNEED),
fallocate(FALLOC_FL_PUNCH_HOLE))
So instead of using anonymous memory+mbind, you can also mmap a sparse file
and rely on populate-on-demand. One alternative for your use case would be
to create a DAX filesystem on that CXL memory (IIRC that should work) and
simply providing virtio-mem with a sparse file located on that filesystem.
Of course, you can also use some other mechanism as you might have in
your approach, as long as it supports a,b,c.
b) (optional) allocate huge pages on that separate NUMA node.
c) Use ordinary memory-device-ram or memory-device-memfd (for huge pages),
*bidning* the memory backend to that special NUMA node.
"-object memory-backend/device-ram or memory-device-memfd, id=mem0, size=768G"
How to bind backend memory to NUMA node
I think the syntax is "policy=bind,host-nodes=X"
whereby X is a node mask. So for node "0" you'd use "host-nodes=0x1" for "5"
"host-nodes=0x20" etc.
This will dynamically allocate memory from that special NUMA node, resulting
in the virtio-mem device completely being backed by that device memory,
being able to dynamically resize the memory allocation.
Exposing an actual devdax to the virtio-mem device, shared by multiple VMs
isn't really what we want and won't work without major design changes. Also,
I'm not so sure it's a very clean design: exposing memory belonging to other
VMs to unrelated QEMU processes. This sounds like a serious security hole:
if you managed to escalate to the QEMU process from inside the VM, you can
access unrelated VM memory quite happily. You want an abstraction
in-between, that makes sure each VM/QEMU process only sees private memory:
for example, the buddy via dax/kmem.
Hi David
Thanks for your suggestion, also sorry for my delayed reply due to my long
vacation.
How does current virtio-mem dynamically attach memory to guest, via page fault?
Essentially you have a large sparse mmap. Withing that mmap, memory is
populated on demand. Instead if mmap/munmap you perform a single large
mmap and then dynamically populate memory/discard memory.
Right now, memory is populated via page faults on access. This is
sub-optimal when dealing with limited resources (i.e., hugetlbfs,
file blocks) and you might run out of backend memory.
I'm working on a "prealloc" mode, which will preallocate/populate memory
necessary for exposing the next block of memory to the VM, and which
fails gracefully if preallocation/population fails in the case of such
limited resources.
The patch resides on:
https://github.com/davidhildenbrand/qemu/tree/virtio-mem-next
commit ded0e302c14ae1b68bdce9059dcca344e0a5f5f0
Author: David Hildenbrand <david@redhat.com>
Date: Mon Aug 2 19:51:36 2021 +0200
virtio-mem: support "prealloc=on" option
Especially for hugetlb, but also for file-based memory backends, we'd
like to be able to prealloc memory, especially to make user errors less
severe: crashing the VM when there are not sufficient huge pages around.
A common option for hugetlb will be using "reserve=off,prealloc=off" for
the memory backend and "prealloc=on" for the virtio-mem device. This
way, no huge pages will be reserved for the process, but we can recover
if there are no actual huge pages when plugging memory.
Signed-off-by: David Hildenbrand <david@redhat.com>
--
Thanks,
David / dhildenb