Re: [PATCH v8 15/15] hw/vmapple/vmapple: Add vmapple machine type

[Phil Dennis-Jordan]

On Mon, 11 Nov 2024 at 05:42, Akihiko Odaki <akihiko.odaki@daynix.com> wrote:
>
> On 2024/11/11 5:18, Phil Dennis-Jordan wrote:
> > I've applied the majority of this feedback in the soon-to-be-posted v9
> > of the patch set, thanks!
> >
> > Just a few extra comments/replies below:
> >
> > On Sun, 10 Nov 2024 at 08:38, Akihiko Odaki <akihiko.odaki@daynix.com> 
> > wrote:
> >>
> >> On 2024/11/08 23:47, Phil Dennis-Jordan wrote:
> >>> From: Alexander Graf <graf@amazon.com>
> >>>
> >>> Apple defines a new "vmapple" machine type as part of its proprietary
> >>> macOS Virtualization.Framework vmm. This machine type is similar to the
> >>> virt one, but with subtle differences in base devices, a few special
> >>> vmapple device additions and a vastly different boot chain.
> >>>
> >>> This patch reimplements this machine type in QEMU. To use it, you
> >>> have to have a readily installed version of macOS for VMApple,
> >>> run on macOS with -accel hvf, pass the Virtualization.Framework
> >>> boot rom (AVPBooter) in via -bios, pass the aux and root volume as pflash
> >>> and pass aux and root volume as virtio drives. In addition, you also
> >>> need to find the machine UUID and pass that as -M vmapple,uuid= parameter:
> >>>
> >>> $ qemu-system-aarch64 -accel hvf -M vmapple,uuid=0x1234 -m 4G \
> >>>       -bios 
> >>> /System/Library/Frameworks/Virtualization.framework/Versions/A/Resources/AVPBooter.vmapple2.bin
> >>>       -drive file=aux,if=pflash,format=raw \
> >>>       -drive file=root,if=pflash,format=raw \
> >>>       -drive file=aux,if=none,id=aux,format=raw \
> >>>       -device vmapple-virtio-aux,drive=aux \
> >>>       -drive file=root,if=none,id=root,format=raw \
> >>>       -device vmapple-virtio-root,drive=root
> >>>
> >>> With all these in place, you should be able to see macOS booting
> >>> successfully.
> >>>
> >>> Known issues:
> >>>    - Keyboard and mouse/tablet input is laggy. The reason for this is
> >>>      either that macOS's XHCI driver is broken when the device/platform
> >>>      does not support MSI/MSI-X, or there's some unfortunate interplay
> >>>      with Qemu's XHCI implementation in this scenario.
> >>>    - Currently only macOS 12 guests are supported. The boot process for
> >>>      13+ will need further investigation and adjustment.
> >>>
> >>> Signed-off-by: Alexander Graf <graf@amazon.com>
> >>> Co-authored-by: Phil Dennis-Jordan <phil@philjordan.eu>
> >>> Signed-off-by: Phil Dennis-Jordan <phil@philjordan.eu>
> >>> ---
> >>>
> >>> v3:
> >>>    * Rebased on latest upstream, updated affinity and NIC creation
> >>>      API usage
> >>>    * Included Apple-variant virtio-blk in build dependency
> >>>    * Updated API usage for setting 'redist-region-count' array-typed 
> >>> property
> >>>      on GIC.
> >>>    * Switched from virtio HID devices (for which macOS 12 does not contain
> >>>      drivers) to an XHCI USB controller and USB HID devices.
> >>>
> >>> v4:
> >>>    * Fixups for v4 changes to the other patches in the set.
> >>>    * Corrected the assert macro to use
> >>>    * Removed superfluous endian conversions corresponding to cfg's.
> >>>    * Init error handling improvement.
> >>>    * No need to select CPU type on TCG, as only HVF is supported.
> >>>    * Machine type version bumped to 9.2
> >>>    * #include order improved
> >>>
> >>> v5:
> >>>    * Fixed memory reservation for ecam alias region.
> >>>    * Better error handling setting properties on devices.
> >>>    * Simplified the machine ECID/UUID extraction script and actually 
> >>> created a
> >>>      file for it rather than quoting its code in documentation.
> >>>
> >>> v7:
> >>>    * Tiny error handling fix, un-inlined function.
> >>>
> >>> v8:
> >>>    * Use object_property_add_uint64_ptr rather than defining custom UUID
> >>>      property get/set functions.
> >>>
> >>>    MAINTAINERS                 |   1 +
> >>>    contrib/vmapple/uuid.sh     |   9 +
> >>>    docs/system/arm/vmapple.rst |  60 ++++
> >>>    docs/system/target-arm.rst  |   1 +
> >>>    hw/vmapple/Kconfig          |  20 ++
> >>>    hw/vmapple/meson.build      |   1 +
> >>>    hw/vmapple/vmapple.c        | 638 ++++++++++++++++++++++++++++++++++++
> >>>    7 files changed, 730 insertions(+)
> >>>    create mode 100755 contrib/vmapple/uuid.sh
> >>>    create mode 100644 docs/system/arm/vmapple.rst
> >>>    create mode 100644 hw/vmapple/vmapple.c
> >>>
> >>> diff --git a/MAINTAINERS b/MAINTAINERS
> >>> index 9d0d26cb65f..9591fd44a34 100644
> >>> --- a/MAINTAINERS
> >>> +++ b/MAINTAINERS
> >>> @@ -2767,6 +2767,7 @@ R: Phil Dennis-Jordan <phil@philjordan.eu>
> >>>    S: Maintained
> >>>    F: hw/vmapple/*
> >>>    F: include/hw/vmapple/*
> >>> +F: docs/system/arm/vmapple.rst
> >>>
> >>>    Subsystems
> >>>    ----------
> >>> diff --git a/contrib/vmapple/uuid.sh b/contrib/vmapple/uuid.sh
> >>> new file mode 100755
> >>> index 00000000000..956e8c3afed
> >>> --- /dev/null
> >>> +++ b/contrib/vmapple/uuid.sh
> >>> @@ -0,0 +1,9 @@
> >>> +#!/bin/sh
> >>> +# Used for converting a guest provisioned using Virtualization.framework
> >>> +# for use with the QEMU 'vmapple' aarch64 machine type.
> >>> +#
> >>> +# Extracts the Machine UUID from Virtualization.framework VM JSON file.
> >>> +# (as produced by 'macosvm', passed as command line argument)
> >>> +
> >>> +plutil -extract machineId raw "$1" | base64 -d | plutil -extract ECID 
> >>> raw -
> >>> +
> >>> diff --git a/docs/system/arm/vmapple.rst b/docs/system/arm/vmapple.rst
> >>> new file mode 100644
> >>> index 00000000000..6a634fa4572
> >>> --- /dev/null
> >>> +++ b/docs/system/arm/vmapple.rst
> >>> @@ -0,0 +1,60 @@
> >>> +VMApple machine emulation
> >>> +========================================================================================
> >>> +
> >>> +VMApple is the device model that the macOS built-in hypervisor called 
> >>> "Virtualization.framework"
> >>> +exposes to Apple Silicon macOS guests. The "vmapple" machine model in 
> >>> QEMU implements the same
> >>> +device model, but does not use any code from Virtualization.Framework.
> >>> +
> >>> +Prerequisites
> >>> +-------------
> >>> +
> >>> +To run the vmapple machine model, you need to
> >>> +
> >>> + * Run on Apple Silicon
> >>> + * Run on macOS 12.0 or above
> >>> + * Have an already installed copy of a Virtualization.Framework macOS 12 
> >>> virtual machine. I will
> >>> +   assume that you installed it using the macosvm CLI.
> >>
> >> Add a URL to macosvm.
> >>
> >>> +
> >>> +First, we need to extract the UUID from the virtual machine that you 
> >>> installed. You can do this
> >>> +by running the shell script in contrib/vmapple/uuid.sh on the 
> >>> macosvm.json file.
> >>> +
> >>> +.. code-block:: bash
> >>> +  :caption: uuid.sh script to extract the UUID from a macosvm.json file
> >>> +
> >>> +  $ contrib/vmapple/uuid.sh "path/to/macosvm.json"
> >>> +
> >>> +Now we also need to trim the aux partition. It contains metadata that we 
> >>> can just discard:
> >>> +
> >>> +.. code-block:: bash
> >>> +  :caption: Command to trim the aux file
> >>> +
> >>> +  $ dd if="aux.img" of="aux.img.trimmed" bs=$(( 0x4000 )) skip=1
> >>
> >> Quoting is inconsistent. aux.img.trimmed is not quoted below but it is
> >> quoted here.
> >>
> >>> +
> >>> +How to run
> >>> +----------
> >>> +
> >>> +Then, we can launch QEMU with the Virtualization.Framework pre-boot 
> >>> environment and the readily
> >>> +installed target disk images. I recommend to port forward the VM's ssh 
> >>> and vnc ports to the host
> >>> +to get better interactive access into the target system:
> >>> +
> >>> +.. code-block:: bash
> >>> +  :caption: Example execution command line
> >>> +
> >>> +  $ UUID=$(uuid.sh macosvm.json)
> >>> +  $ 
> >>> AVPBOOTER=/System/Library/Frameworks/Virtualization.framework/Resources/AVPBooter.vmapple2.bin
> >>> +  $ AUX=aux.img.trimmed
> >>> +  $ DISK=disk.img
> >>> +  $ qemu-system-aarch64 \
> >>> +       -serial mon:stdio \
> >>> +       -m 4G \
> >>> +       -accel hvf \
> >>> +       -M vmapple,uuid=$UUID \
> >>> +       -bios $AVPBOOTER \
> >>
> >> $AUX and $DISK are quoted but $UUID and $AVPBOOTER are not.
> >>
> >>> +        -drive file="$AUX",if=pflash,format=raw \
> >>> +        -drive file="$DISK",if=pflash,format=raw \
> >>
> >> These two lines are misaligned.
> >>
> >>> +       -drive file="$AUX",if=none,id=aux,format=raw \
> >>> +       -drive file="$DISK",if=none,id=root,format=raw \
> >>> +       -device vmapple-virtio-aux,drive=aux \
> >>> +       -device vmapple-virtio-root,drive=root \
> >>> +       -net user,ipv6=off,hostfwd=tcp::2222-:22,hostfwd=tcp::5901-:5900 \
> >>> +       -net nic,model=virtio-net-pci \
> >>
> >> -net is a legacy option and creates a unnecessary indirection with hub.
> >> Use -netdev and -device options. Also plug virtio-net-pci into a PCIe
> >> root port as suggested in: docs/pcie.txt
> >
> > I have not had any success with booting macOS with the block devices
> > connected to PCIe root ports, and similarly the network doesn't come
> > up if I attach just the virtio-net-pci device to one.
> >
> > As far as I can tell (ioreg), Apple's own VMapple machine type does
> > not use PCIe root ports, but I'm not 100% sure. The PCIe documentation
> > is generally rather focused on x86 and on running Linux in the 'virt'
> > machine type.
>
> Without a PCIe root port, virtio devices will be forced to work as a PCI
> device (not PCIe) and to use its "legacy" interface. I don't think
> Virtualization.framework uses the legacy interface though.

I'm not sure that's true, the GPEX controller purports to support PCIe
in some form. And even if it is falling back to legacy Virtio mode,
the guest OS doesn't appear to care. The guest's view of the PCI(e?)
setup looks extremely similar between Virtualization.framework and the
Qemu vmapple machine type we're adding here. I've attached `ioreg`
output for some relevant I/O registry node types for both Qemu
(vmapple-qemu-ioreg.txt) and macosvm (vmapple-vzfwk-ioreg.txt).

> > In any case, a far more pressing issue is MSI support, without which
> > the XHCI driver fails to receive most kinds of notifications from the
> > USB controller. (Therefore keyboard/mouse is currently very laggy.)
> > The arm 'virt' machine type supposedly supports it via the GIC ITS
> > device, but I've so far not managed to transfer this to 'vmapple' in a
> > way that makes the guest OS detect MSI support. Anyway, in its current
> > iteration, the machine type is good enough to be used for some tasks,
> > and we can always refine it later. (Another major limitation is of
> > course guest support for OS versions > 12 - but those issues might
> > even be related.)
>
> Perhaps the MSI issue may be also related to PCIe.

MSI is certainly intended to be the "native" method for implementing
interrupts on PCIe. However:

Reviewing the ioreg output, I actually see no evidence that
Virtualization.framework is even using MSIs. In particular, take note
of the IOInterruptControllers property on each IOPCIDevice. In both
VZ.fwk and Qemu vmapple, the Virtio and XHCI devices use this:

                |   "IOInterruptControllers" = ("IOInterruptController00000037")

This in turn simply corresponds to the GIC:

            +-o AppleARMGICv3  <class AppleARMGICv3, id 0x10000016b,
registered, matched, active, busy 0 (4 ms), retain 6>
                {
…
                  "InterruptControllerName" = "IOInterruptController00000037"
…

On a physical Mac, or an x86-64 VM, there will additionally be a
separate interrupt controller object for MSIs, which is a subclass of
IOPCIMessagedInterruptController.

So perhaps the problem isn't lack of MSIs after all (the guest even
may not support them) but rather that Qemu's XHCI controller doesn't
implement the case of 1 interrupter in a way that macOS's XHCI driver
supports.

(Qemu's XHCI controller defaults to 16 interrupters, with 1 MSI vector
per interrupter. When MSIs are unavailable, numintrs remains 16 even
though the legacy interrupt mechanism only provides a single IRQ line,
and when the XHCI controller tries to signal vector 1 or higher, the
guest isn't notified. Unfortunately, a quick hack that detects lack of
MSI support and sets numintrs to 1 makes things even worse. I've tried
this with an x86-64 VM as well, and the macOS guest stops being able
to drive the XHCI controller correctly when I force numintrs to 1.

Linux guests were able to cope with it if I remember correctly though,
so I'm not sure where exactly the problem lies:
 A. Does Qemu implement numintrs=1 badly, but Linux's driver is
tolerant enough to make it work regardless?
 B. Or does macOS's XHCI driver expect a very specific device
behaviour with numintrs=1, which doesn't match Qemu's?
Either way, we can't fix it on the guest side here…

Interestingly, Virtualization.framework also exposes an
AppleVirtIOUSBDeviceController to the guest. This isn't used for the
virtual keyboard and digitiser though, and I've not found any protocol
information on USB-over-Virtio, so who knows if that would be in any
way useful as a generic USB controller.)

Long story short: I think we're best off leaving the PCI(e) setup as
it is for the moment until we figure out exactly what's needed to fix
the XHCI controller.

> > One thing I am thinking about changing (as we've now missed the 9.2
> > merge window anyway, sigh) as it would be non-backwards-compatible is
> > replacing the vmapple-virtio-aux/vmapple-virtio-root pair of devices
> > with just the the base type, vmapple-virtio-blk-pci, and adding a
> > variant property to it. So:
> >
> >         -device vmapple-virtio-blk-pci,variant=aux,drive=aux \
> >         -device vmapple-virtio-blk-pci,variant=root,drive=root \
> >
> > I think this would make hw/vmapple/virtio-blk.c somewhat more readable
> > and also significantly shorter.
>
> I agree. It will be a nice improvement.

vmapple-vzfwk-ioreg.txt
Description: Text document

vmapple-qemu-ioreg.txt
Description: Text document