Re: [Qemu-devel] [PULL 22/25] target/i386: kvm: Add nested migration blocker only when kernel lacks required capabilities

[Jan Kiszka]

On 21.06.19 13:30, Paolo Bonzini wrote:
> From: Liran Alon <address@hidden>
> 
> Previous commits have added support for migration of nested virtualization
> workloads. This was done by utilising two new KVM capabilities:
> KVM_CAP_NESTED_STATE and KVM_CAP_EXCEPTION_PAYLOAD. Both which are
> required in order to correctly migrate such workloads.
> 
> Therefore, change code to add a migration blocker for vCPUs exposed with
> Intel VMX or AMD SVM in case one of these kernel capabilities is
> missing.
> 
> Signed-off-by: Liran Alon <address@hidden>
> Reviewed-by: Maran Wilson <address@hidden>
> Message-Id: <address@hidden>
> Signed-off-by: Paolo Bonzini <address@hidden>
> ---
>  target/i386/kvm.c     | 9 +++++++--
>  target/i386/machine.c | 2 +-
>  2 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/target/i386/kvm.c b/target/i386/kvm.c
> index c931e9d..e4b4f57 100644
> --- a/target/i386/kvm.c
> +++ b/target/i386/kvm.c
> @@ -1640,9 +1640,14 @@ int kvm_arch_init_vcpu(CPUState *cs)
>                                    !!(c->ecx & CPUID_EXT_SMX);
>      }
>  
> -    if (cpu_has_nested_virt(env) && !nested_virt_mig_blocker) {
> +    if (cpu_has_vmx(env) && !nested_virt_mig_blocker &&
> +        ((kvm_max_nested_state_length() <= 0) || !has_exception_payload)) {
>          error_setg(&nested_virt_mig_blocker,
> -                   "Nested virtualization does not support live migration 
> yet");
> +                   "Kernel do not provide required capabilities for "
> +                   "nested virtualization migration. "
> +                   "(CAP_NESTED_STATE=%d, CAP_EXCEPTION_PAYLOAD=%d)",
> +                   kvm_max_nested_state_length() > 0,
> +                   has_exception_payload);
>          r = migrate_add_blocker(nested_virt_mig_blocker, &local_err);
>          if (local_err) {
>              error_report_err(local_err);
> diff --git a/target/i386/machine.c b/target/i386/machine.c
> index fc49e5a..851b249 100644
> --- a/target/i386/machine.c
> +++ b/target/i386/machine.c
> @@ -233,7 +233,7 @@ static int cpu_pre_save(void *opaque)
>  
>  #ifdef CONFIG_KVM
>      /* Verify we have nested virtualization state from kernel if required */
> -    if (cpu_has_nested_virt(env) && !env->nested_state) {
> +    if (kvm_enabled() && cpu_has_vmx(env) && !env->nested_state) {
>          error_report("Guest enabled nested virtualization but kernel "
>                  "does not support saving of nested state");
>          return -EINVAL;
> 

Starting with this commit latest (bisection issue...), running Jailhouse in a
guest first stalls L1 (looks like we lose interrupts), and if I try to reset
that VM, I lose my host as well:

kvm: vmptrld           (null)/6eb900000000 failed
kvm: vmclear fail:           (null)/6eb900000000

and then things start to lock up because we seem to lose the CPUs the guest was
running on. Once I had this in the logs:

rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 15040
jiffies s: 4673 root: 0x80/.
rcu: blocking rcu_node structures:
Task dump for CPU 7:
qemu-system-x86 R  running task        0 17413  17345 0x00000008
Call Trace:
 ? x86_virt_spec_ctrl+0x7/0xe0
 ? vmx_vcpu_run.part.0+0x2a4/0xfa0 [kvm_intel]
 ? vcpu_enter_guest+0x349/0xe80 [kvm]
 ? kvm_arch_vcpu_ioctl_run+0xff/0x550 [kvm]
 ? kvm_vcpu_ioctl+0x20d/0x590 [kvm]
 ? get_futex_key+0x35d/0x3b0
 ? do_vfs_ioctl+0x447/0x640
 ? do_futex+0x157/0x1d0
 ? ksys_ioctl+0x5e/0x90
 ? __x64_sys_ioctl+0x16/0x20
 ? do_syscall_64+0x60/0x120
 ? entry_SYSCALL_64_after_hwframe+0x49/0xbe

This was on a 5.1.16 distro kernel. Currently rebuilding 5.2 vanilla.

Looks like we have up to two critical bugs here...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux