Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?

[Peter Xu]

On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote:
> Hi,

Hello, Jintack,

> 
> I'm trying to pass through a physical network device to a nested VM
> using virtual IOMMU. While I was able to do it successfully using KVM
> and Xen guest hypervisors running in a VM respectively, I couldn't do
> it with Hyper-V as I described below. I wonder if anyone have
> successfully used virtual IOMMU in other hypervisors other than KVM
> and Xen? (like Hyper-V or VMware)
> 
> The issue I have with Hyper-V is that Hyper-V gives an error that the
> underlying hardware is not capable of doing passthrough. The exact
> error message is as follows.
> 
> Windows Power-shell > (Get-VMHost).IovSupportReasons
> The chipset on the system does not do DMA remapping, without which
> SR-IOV cannot be supported.
> 
> I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have
> enabled iommu in windows boot loader[1], and I see differences when
> booing a Windows VM with and without virtual IOMMU. I also checked
> that virtual IOMMU traces are printed.

What traces have you checked?  More explicitly, have you seen DMAR
enabled and page table setup for that specific device to be
pass-throughed?

> 
> I have tried multiple KVM/QEMU versions including the latest ones
> (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers
> (2016 and 2019), but I see the same result. [4]
> 
> I'd love to hear if somebody is using virtual IOMMU in Hyper-V or
> VMware successfully, especially for passthrough. I also appreciate if
> somebody can point out any configuration errors I have.
> 
> Here's the qemu command line I use, basically from the QEMU vt-d
> page[2] and Hyper-v on KVM from kvmforum [3].
> 
> ./qemu/x86_64-softmmu/qemu-system-x86_64 -device
> intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M

Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest
QEMU, or x-aw-bits=48 on some old ones)?  IIRC we've encountered
issues when trying to pass the SVVP Windows test with this, in which
4-level is required.  I'm not sure whether whether that is required in
general usages of vIOMMU in Windows.

> q35,accel=kvm,kernel-irqchip=split -cpu
> host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -drive
> if=none,file=/vm/guest0.img,id=vda,cache=none,format=raw -device
> virtio-blk-pci,drive=vda --nographic -qmp
> unix:/var/run/qmp,server,nowait -serial
> telnet:127.0.0.1:4444,server,nowait -netdev
> user,id=net0,hostfwd=tcp::2222-:22 -device
> virtio-net-pci,netdev=net0,mac=de:ad:be:ef:f2:12 -netdev
> tap,id=net1,vhost=on,helper=/srv/vm/qemu/qemu-bridge-helper -device
> virtio-net-pci,netdev=net1,disable-modern=off,disable-legacy=on,mac=de:ad:be:ef:f2:11
> -device vfio-pci,host=0000:06:10.0,id=net2 -monitor stdio -usb -device
> usb-tablet -rtc base=localtime,clock=host -vnc 127.0.0.1:4 --cdrom
> win19.iso --drive file=virtio-win.iso,index=3,media=cdrom

-- 
Peter Xu