Re: [Qemu-discuss] Mutil gpu passthrough failed.

[John Y.]

I get it, thank you very much.

Regards,

John

2017-01-11 3:54 GMT+08:00 Aleksei <address@hidden>:

No. Just tried win7 iso on fresh VM with UEFI - got to "Install Windows" screen without issues. Not using VNC though.

My software is libvirt 2.4.0, qemu 2.8.0.

--Regards, Aleksei

---

From: John Y.
Sent: Tuesday, January 10, 2017 1:41PM
To: Aleksei
Cc: Qemu-discuss
Subject: Re: [Qemu-discuss] Mutil gpu passthrough failed.

What about the vm image? Install from iso  or covert a existing VM from BIOS to UEFI ?

I got a black screen in vnc after VM load the iso file and 'starting windows' when I install W7 by iso and boot with UEFI.

Have you ever encountered this situation?

Regards,

John

2017-01-10 18:33 GMT+08:00 John Y. <address@hidden>:

What about the vm image? Install from iso  or covert a existing VM from BIOS to UEFI ?

Regards,

John

2017-01-10 18:14 GMT+08:00 Aleksei <address@hidden>:

I also use VARS part of OVMF:

    <loader readonly='yes' type='pflash'>/vms/UEFI_OVMF_win7/OVMF_CODE.fd</loader>
    <nvram>/home/username/.config/libvirt/qemu/nvram/win7_VARS.fd</nvram>

You might need to compile OVMF yourself to get both CODE and VARS parts. Not sure how it's handled in CentOS, here's an explanation for Arch https://wiki.archlinux.org/index.php/Libvirt#OVMF_-_QEMU_workaround

--Regards, Aleksei

---

From: John Y.
Sent: Tuesday, January 10, 2017 1:05PM
To: Aleksei
Cc: Qemu-discuss
Subject: Re: [Qemu-discuss] Mutil gpu passthrough failed.

I add <loader type='pflash'>/root/yaowj/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> to libvirt and I want to install W7, buy vm hangs after loading the iso.

I searched google and someone said that windows 7 is not support UEFI completely.

Can you give me a link about libvirt/qemu/OVMF/Windows7 ? How do you make the W7 image?

Regards

John

2017-01-10 17:57 GMT+08:00 Aleksei <address@hidden>:

> UEFI is unfriendly to windows7

What do you mean? I use libvirt/qemu/OVMF/Windows7 combination all the time, W7 works fine with OVMF.

--Regards, Aleksei

---

From: John Y.
Sent: Tuesday, January 10, 2017 12:50PM
To: Aleksei
Cc: Qemu-discuss
Subject: Re: [Qemu-discuss] Mutil gpu passthrough failed.

I sorry that I give the wrong info in last email and here is  the IOMMU Group about 0000:41:00.0 :

IOMMU Group 40 41:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710] [10de:1281] (rev a1)

IOMMU Group 40 41:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

I have other test in this case:

I have two graphics card bios rom files:

1.  One ( rom A ) is export by  :

     # echo 1 > /sys/bus/pci/devices/0000\:41\:00.0/rom

     # cat /sys/bus/pci/devices/0000\:41\:00.0/rom > /root/nvidia/gt710.rom

     # echo 0 > /sys/bus/pci/devices/0000\:41\:00.0/rom

     This is which I use now and mention in my first email.

2.  Aother ( rom B ) is exported by gpu-z in windows 7 : /root/nvidia/gpuz.rom

As I write in my first email.

VM had no output to screen and I can not ping the VM with /root/nvidia/gt710.rom. (Which works fine when I just pass group A or group B to vm.)

And the newest test results are :

(VM use 0000:41:00.0 while another using 0000:05:00.0)

1. VM work fine when I use rom B and installed cento7 to VM .

2. VM had no output to then screen, but I can ping the VM  when I use rom B and installed window7 to VM,  the gpu return code 43  in Device Manager. (Is that mean I use a wrong rom file ? )

3. VM work fine when I start VM with ovmf  in windows 10 (boot from UEFI).

And here are my questions are:

1. Is this problems about the rom file of gpu?

If the answer is 'yes' and :

2. How can I get the correct and useful rom file ? I want to use window7 in guest, but UEFI is unfriendly to windows7.

3. Why I can start my vm when I pass group B only and use rom A, but I can not start vm when I pass group A and B to vms and also use rom A ?

Regards

John

 

2017-01-10 15:57 GMT+08:00 Aleksei <address@hidden>:

What about the other two, maybe they are in one group?

Also, you had these addresses in your first email - 0000:04:00.0, 0000:05:00.0, 0000:41:00.0, 0000:42:00.0.0 and now you have a GPU at 44:00.0, are you changing physical allocation as well?

--Regards, Aleksei

---

From: John Y.
Sent: Tuesday, January 10, 2017 3:44AM
To: Aleksei
Cc: Qemu-discuss
Subject: Re: [Qemu-discuss] Mutil gpu passthrough failed.

Hi Aleksei, 

Thasks for your help.

The PCIe slots is isolate:

./check_iommu.sh  | grep IOMMU\ Group\ 41

IOMMU Group 41 44:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710] [10de:1281] (rev a1)

IOMMU Group 41 44:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

# ./check_iommu.sh  | grep IOMMU\ Group\ 19

IOMMU Group 19 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710] [10de:1281] (rev a1)

IOMMU Group 19 05:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

I think the vm can not boot if I just pass group B to vm when system is putting some PCIe slots in IOMMU groups.

Any ideas?

2017-01-09 23:34 GMT+08:00 Aleksei <address@hidden>:

Check if your system is putting some PCIe slots in IOMMU groups, see explanation here: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot

--Regards, Aleksei

---

From: John Y.
Sent: Monday, January 09, 2017 5:50PM
To: Qemu-discuss
Subject: [Qemu-discuss] Mutil gpu passthrough failed.

My server has 4 PCI slots, their addresses are 0000:04:00.0, 0000:05:00.0, 0000:41:00.0, 0000:42:00.0.

I want to pass 4 gpus to vms via vfio, each of vms has one gpu.

Vms worked fine when I passed 0000:04:00.0 and 0000:05:00.0 to vms or passed 0000:41:00.0 and 0000:42:00.0 to vms.

But vms which used 0000:41:00.0 and 0000:42:00.0 had no output to the screen and I could not ping vm's ip when I passed 0000:04:00, 0000:41:00.0 and 0000:42:00.0 to vms, but vm which used 0000:04:00 worked fine.

I got the same result when I passed 0000:05:00, 0000:41:00.0 and 0000:42:00.0 to vms :  vm with 0000:05:00 worked fine and vms which used 0000:41:00.0 and 0000:42:00.0 hanged.

When I define 0000:04:00.0 and 0000:05:00.0 as group A, 0000:41:00.0 and 0000:42:00.0 as group B, the result is :

1. Vms work fine When I just pass group A or group B to vms.

2. Just group A works fine when the gpus that I pass to vms contain group A and B.

In case 2, vms work fine when I used '-vga cirrus' instead of '-vga none'. But I get code 12 from graphics card in Device Manager of windows 7. And this is not what I want, I just want to use the graphics card in the host, so I should use '-vga none'.

It also worked fine when I remove vga passthrough or just pass 0000:41:00.1(the hdmi audio of graphics card) to vm.

I got no error output in qemu monitor.

How can I solve this probem ?

Version of qemu:

QEMU emulator version 2.5.1.1, Copyright (c) 2003-2008 Fabrice Bellard

Qemu command:

/root/qemu25/qemu-system-x86_64 \

-name gputest_41 \

-machine q35,accel=kvm,usb=off,smm=off \

-cpu host \

-m 4096 \

-realtime mlock=off \

-smp 4,sockets=1,cores=4,threads=1 \

-no-user-config \

-nodefaults  \

-rtc base=localtime \

-no-shutdown \

-boot strict=on \

-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \

-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 \

-drive file=/root/gpu/gputest_41/win7.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 \

-device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \

-k en-us \

-device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x4 \

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \

-device vfio-pci,host=41:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/nvidia/gt710.rom,x-vga=on \

-msg timestamp=on \

-vga none \

-cpu host,hv_vendor_id=1234567890ab,kvm=off \

-net nic,model=virtio \

-net tap,ifname=tap1,script=/root/qemu/qemu-ifup.sh,downscript=/root/qemu/qemu-ifdown.sh \

-monitor stdio 

System info:

Linux ns.mqcache.net 4.2.0-1.el7.elrepo.x86_64 #1 SMP Sun Aug 30 21:25:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

Output of lspci -vvv:

05:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])

Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 15

NUMA node: 0

Region 0: Memory at d8000000 (32-bit, non-prefetchable) [size=16M]

Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]

Region 3: Memory at ce000000 (64-bit, prefetchable) [size=32M]

Region 5: I/O ports at ec80 [size=128]

Expansion ROM at d9000000 [disabled] [size=512K]

Capabilities: [60] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

Address: 0000000000000000  Data: 0000

Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us

ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-

Capabilities: [100 v1] Virtual Channel

Caps: LPEVC=0 RefClk=100ns PATEntryBits=1

Arb: Fixed- WRR32- WRR64- WRR128-

Ctrl: ArbSelect=Fixed

Status: InProgress-

VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

Status: NegoPending- InProgress-

Capabilities: [128 v1] Power Budgeting <?>

Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>

Capabilities: [900 v1] #19

Kernel driver in use: vfio-pci

Kernel modules: nouveau

05:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)

Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin B routed to IRQ 14

NUMA node: 0

Region 0: Memory at d9ffc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [60] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

Address: 0000000000000000  Data: 0000

Capabilities: [78] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us

ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Kernel driver in use: vfio-pci

Kernel modules: snd_hda_intel

41:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])

Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929

Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Interrupt: pin A routed to IRQ 15

NUMA node: 1

Region 0: Memory at b8000000 (32-bit, non-prefetchable) [disabled] [size=16M]

Region 1: Memory at 98000000 (64-bit, prefetchable) [disabled] [size=128M]

Region 3: Memory at a6000000 (64-bit, prefetchable) [disabled] [size=32M]

Region 5: I/O ports at bc80 [disabled] [size=128]

Expansion ROM at b9000000 [disabled] [size=512K]

Capabilities: [60] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

Address: 0000000000000000  Data: 0000

Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us

ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-

Capabilities: [100 v1] Virtual Channel

Caps: LPEVC=0 RefClk=100ns PATEntryBits=1

Arb: Fixed- WRR32- WRR64- WRR128-

Ctrl: ArbSelect=Fixed

Status: InProgress-

VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff

Status: NegoPending- InProgress-

Capabilities: [128 v1] Power Budgeting <?>

Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>

Capabilities: [900 v1] #19

Kernel driver in use: vfio-pci

Kernel modules: nouveau

41:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)

Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin B routed to IRQ 14

NUMA node: 1

Region 0: Memory at b9ffc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [60] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+

Address: 0000000000000000  Data: 0000

Capabilities: [78] Express (v2) Endpoint, MSI 00

DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W

DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+

RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+

MaxPayload 256 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us

ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported

DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled

LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Kernel driver in use: vfio-pci

Kernel modules: snd_hda_intel

After starting vm:

lspci -vvv

05:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])

Subsystem: Micro-Star International Co., Ltd. [MSI] Device 2929

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 102

NUMA node: 0

Region 0: Memory at d8000000

...

[邮件部分隐藏]