Re: [PATCH][RESEND v5 3/3] Add a Hyper-V Dynamic Memory Protocol driver (hv-balloon)

[David Hildenbrand]

On 22.06.23 13:17, Maciej S. Szmigiero wrote:
> On 22.06.2023 13:15, David Hildenbrand wrote:
> > On 22.06.23 13:12, Maciej S. Szmigiero wrote:
> > > On 22.06.2023 13:01, David Hildenbrand wrote:
> > > > [...]
> > > >
> > > > > > > > We'd use a memory region container as device memory region (like [1]) and would 
> > > > > > > > have to handle the !memdev case (I can help with that). > Into that, you can 
> > > > > > > > map the RAM memory region on demand (and eventually even using multiple slots like 
> > > > > > > > [1]).
> > > > > > > >
> > > > > > > > (2) Use a single virtual DIMM and (un)plug that on demand. Let the machine code 
> > > > > > > > handle (un)plugging of the device.
> > > > > > > >
> > > > > > > >
> > > > > > > > (1) feels cleanest to me, although it will require a bit more work.
> > > > > > >
> > > > > > > I also think approach (1) makes more sense as it avoids memslot metadata
> > > > > > > overhead for not-yet-hot-added parts of the memory backing device.
> > > > > > >
> > > > > > > Not sure what you mean that the !memdev case would be problematic in this
> > > > > > > case - it is working in the current driver shape so why would adding
> > > > > > > potential memory subregions (used in the memdev case) change that?
> > > > > >
> > > > > > I'm thinking about the case where you have a hv-balloon device without a memdev.
> > > > > >
> > > > > > Without -m X,maxmem=y we don't currently expect to have memory devices around
> > > > > > (and especially them getting (un)plugged. But why should we "force" to set the
> > > > > > "maxmem" option
> > > > >
> > > > > I guess it's only a small change to QEMU to allow having hv-balloon
> > > > > device (without a memdev) even in the case where there's no "maxmem"
> > > > > option given on the QEMU command line.
> > > > >
> > > > > > I hope I'll find some time soonish to prototype what I have in mind, to see
> > > > > > if it could be made working.
> > > > >
> > > > > Okay, so I'll wait for your prototype before commencing further work on
> > > > > the next version of this driver.
> > > >
> > > > About to have something simplistic running -- I think. Want to test with a 
> > > > Linux VM, but I don't seem to get it working (also without my changes).
> > > >
> > > >
> > > > #!/bin/bash
> > > >
> > > > build/qemu-system-x86_64 \
> > > >        --enable-kvm \
> > > >        -m 4G,maxmem=36G \
> > > >        -cpu host,hv-syndbg=on,hv-synic,hv-relaxed,hv-vpindex \
> > > >        -smp 16 \
> > > >        -nographic \
> > > >        -nodefaults \
> > > >        -net nic -net user \
> > > >        -chardev stdio,nosignal,id=serial \
> > > >        -hda Fedora-Cloud-Base-37-1.7.x86_64.qcow2 \
> > > >        -cdrom /home/dhildenb/git/cloud-init/cloud-init.iso \
> > > >        -device isa-serial,chardev=serial \
> > > >        -chardev socket,id=monitor,path=/var/tmp/mon_src,server,nowait \
> > > >        -mon chardev=monitor,mode=readline \
> > > >        -device vmbus-bridge \
> > > >        -object memory-backend-ram,size=2G,id=mem0 \
> > > >        -device hv-balloon,id=hv1,memdev=mem0
> > > >
> > > >
> > > >
> > > > [root@vm-0 ~]# uname -r
> > > > 6.3.5-100.fc37.x86_64
> > > > [root@vm-0 ~]# modprobe hv_balloon
> > > > modprobe: ERROR: could not insert 'hv_balloon': No such device
> > > >
> > > >
> > > > Any magic flag I am missing? Or is there something preventing this to work with 
> > > > Linux VMs?
> > >
> > > Haven't tested the driver with Linux guests in a long time (as it is
> > > targeting Windows), but I think you need to disable KVM PV interface for
> > > the Hyper-V one to be detected by Linux.
> > >
> > > Something like adding "kvm=off" to "-cpu" and seeing in the dmesg whether
> > > the detected hypervisor is now Hyper-V.
> > >
> > > Also, you need to disable S4 in the guest for hot-add capability to work
> > > (I'm adding "-global ICH9-LPC.disable_s4=1" with q35 machine for this).
> > >
> > > Would also suggest adding "--trace 'hv_balloon_*' --trace 'memory_device_*'"
> > > to QEMU command line to see what's happening.
> >
> > VM is not happy:
> >
> > [    1.908595] BUG: kernel NULL pointer dereference, address: 0000000000000007
> > [    1.908837] #PF: supervisor read access in kernel mode
> > [    1.908837] #PF: error_code(0x0000) - not-present page
> > [    1.908837] PGD 0 P4D 0
> > [    1.908837] Oops: 0000 [#1] PREEMPT SMP NOPTI
> > [    1.908837] CPU: 13 PID: 492 Comm: (udev-worker) Not tainted 
> > 6.3.5-100.fc37.x86_64 #1
> > [    1.908837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
> > rel-1.16.2-0-gea1b7a073390-p4
> > [    1.908837] RIP: 0010:acpi_ns_lookup+0x8f/0x4c0
> > [    1.908837] Code: 8b 3d f5 eb 1c 03 83 05 52 ec 1c 03 01 48 85 ff 0f 84 51 
> > 03 00 00 44 89 c3 4c 89 cb
> > [    1.908837] RSP: 0018:ffff95b680ad7950 EFLAGS: 00010286
> > [    1.908837] RAX: ffff95b680ad79e0 RBX: 0000000000000002 RCX: 0000000000000003
> > [    1.908837] RDX: 0000000000000000 RSI: ffff8a0283a3c558 RDI: ffffffffa4b376e0
> > [    1.908837] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000
> > [    1.908837] R10: ffff8a02811034ec R11: 0000000000000000 R12: ffffffffffffffff
> > [    1.908837] R13: ffff8a02811034e8 R14: ffff8a02811034e8 R15: 0000000000000000
> > [    1.908837] FS:  00007f3bb2e7d0c0(0000) GS:ffff8a02bbd40000(0000) 
> > knlGS:0000000000000000
> > [    1.908837] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [    1.908837] CR2: 0000000000000007 CR3: 0000000100a58002 CR4: 0000000000770ee0
> > [    1.908837] PKRU: 55555554
> > [    1.908837] Call Trace:
> > [    1.908837]  <TASK>
> > [    1.908837]  ? __die+0x23/0x70
> > [    1.908837]  ? page_fault_oops+0x171/0x4e0
> > [    1.908837]  ? prepare_alloc_pages.constprop.0+0xf6/0x1a0
> > [    1.908837]  ? exc_page_fault+0x74/0x170
> > [    1.908837]  ? asm_exc_page_fault+0x26/0x30
> > [    1.908837]  ? acpi_ns_lookup+0x8f/0x4c0
> > [    1.908837]  acpi_ns_get_node_unlocked+0xdd/0x110
> > [    1.908837]  ? down_timeout+0x3e/0x60
> > [    1.908837]  ? acpi_ns_get_node+0x3e/0x60
> > [    1.908837]  acpi_ns_get_node+0x3e/0x60
> > [    1.908837]  acpi_ns_evaluate+0x1cb/0x2d0
> > [    1.908837]  acpi_ut_evaluate_object+0x68/0x1c0
> > [    1.908837]  acpi_rs_get_method_data+0x37/0x80
> > [    1.908837]  ? __pfx_vmbus_walk_resources+0x10/0x10 [hv_vmbus]
> > [    1.908837]  acpi_walk_resources+0x91/0xe0
> > [    1.908837]  vmbus_acpi_add+0x87/0x170 [hv_vmbus]
> > [    1.908837]  acpi_device_probe+0x47/0x160
> > [    1.908837]  really_probe+0x19f/0x400
> > [    1.908837]  ? __pfx___driver_attach+0x10/0x10
> > [    1.908837]  __driver_probe_device+0x78/0x160
> > [    1.908837]  driver_probe_device+0x1f/0x90
> > [    1.908837]  __driver_attach+0xd2/0x1c0
> > [    1.908837]  bus_for_each_dev+0x85/0xd0
> > [    1.908837]  bus_add_driver+0x116/0x220
> > [    1.908837]  driver_register+0x59/0x100
> > [    1.908837]  ? __pfx_init_module+0x10/0x10 [hv_vmbus]
> > [    1.908837]  hv_acpi_init+0x39/0xff0 [hv_vmbus]
> > [    1.908837]  ? __pfx_init_module+0x10/0x10 [hv_vmbus]
> > [    1.908837]  do_one_initcall+0x5a/0x240
> > [    1.908837]  do_init_module+0x4a/0x210
> > [    1.908837]  __do_sys_init_module+0x17f/0x1b0
> > [    1.908837]  do_syscall_64+0x5c/0x90
> > [    1.908837]  ? handle_mm_fault+0x11e/0x310
> > [    1.908837]  ? do_user_addr_fault+0x1e0/0x720
> > [    1.908837]  ? exc_page_fault+0x74/0x170
> > [    1.908837]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> I guess *few* people run Linux with QEMU Hyper-V interfaces
> implementation..
>    
> > Maybe I'll have to install a Windows guest :/
> I think that makes more sense, since we're targeting Windows anyway.

Having installed fairly recent Win10 and running with master+your 
patches, I still can't get it to work.

Windows is stuck booting (before the little circle starts turning).

Removing the hv-balloon device makes it work again (well, at least the 
circle spins again my windows installation now seems to be broken and I 
have to reinstall ... windows).

Do you have a working cmdline for Windows I can try?

--
Cheers,

David / dhildenb