[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[RFC PATCH 00/26] Confidential guest live migration
From: |
Dov Murik |
Subject: |
[RFC PATCH 00/26] Confidential guest live migration |
Date: |
Tue, 2 Mar 2021 15:47:56 -0500 |
This is an RFC series for fast migration of confidential guests using an
in-guest migration helper that lives in OVMF. QEMU VM live migration
needs to read source VM's RAM and write it in the target VM; this
mechanism doesn't work when the guest memory is encrypted or QEMU is
prevented from reading it in another way. In order to support live
migration in such scenarios, we introduce an in-guest migration helper
which can securely extract RAM content from the guest in order to send
it to the target. The migration helper is implemented as part of the
VM's firmware in OVMF.
We've implemented and tested this on AMD SEV, but expect most of the
processes can be used with other technologies that prevent direct access
of hypervisor to the guest's memory. Specifically, we don't use SEV's
PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all.
Corresponding RFC patches for OVMF have been posted by my colleague
Tobin Feldman-Fitzthum on edk2-devel. Those include the crux of the
migration helper: a mailbox protocol over a shared memory page which
allows communication between QEMU and the migration helper. In the
source VM this is used to read a page and encrypt it for transport; in
the target it is used to decrypt the incoming page and storing the
content in the correct address in the guest memory. All encryption and
decryption operations occur inside the trusted context in the VM, and
therefore the VM's memory plaintext content is never accessible to the
hosts participating in the migration.
In order to allow OVMF to run the migration helper in parallel to the
guest OS, we introduce the notion of auxiliary vcpus, which are usable
for OVMF but are hidden from the guest OS. These might have other
future uses for in-guest operations/agents.
In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.
The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not. This is currently achieved using AMD's
patches which track the encryption status of guest pages in KVM, using
hypercalls from OVMF and guest Linux to report changes of such status.
The QEMU side of these patches is included as the first two patches in
this series. The concrete implementation of this page encryption tracking
is currently in flux in the KVM mailing list, but the underlying
implementation doesn't affect our confidential RAM migration as long as
it can check whether a given guest address is encrypted.
List of patches in this series:
1-2: reposting AMD encrypted page bitmap support.
3-11: introduce the notion of auxiliary vcpus.
12-21: introduce the migration specifics.
22-23: fix devices issues when loading state into a live VM
24: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
25: remove SEV migration blocker
26: add documentation
Brijesh Singh (1):
kvm: add support to sync the page encryption state bitmap
Dov Murik (21):
linux-headers: Add definitions of KVM page encryption bitmap ioctls
machine: Add auxcpus=N suboption to -smp
hw/boards: Add aux flag to CPUArchId
hw/i386: Mark auxiliary vcpus in possible_cpus
cpu: Add boolean aux field to CPUState
hw/i386: Set CPUState.aux=true for auxiliary vcpus
softmmu: Don't sync aux vcpus in pre_loadvm
softmmu: Add cpu_synchronize_without_aux_post_init
migration: Add helpers to save confidential RAM
migration: Add helpers to load confidential RAM
migration: Introduce gpa_inside_migration_helper_shared_area
migration: Save confidential guest RAM using migration helper
migration: Load confidential guest RAM using migration helper
migration: Stop VM after loading confidential RAM
migration: Don't sync vcpus when migrating confidential guests
migration: When starting target, don't sync auxiliary vcpus
hw/isa/lpc_ich9: Allow updating an already-running VM
target/i386: Re-sync kvm-clock after confidential guest migration
migration: Add start-migrate-incoming QMP command
target/i386: SEV: Allow migration unless there are no aux vcpus
docs: Add confidential guest live migration documentation
Tobin Feldman-Fitzthum (4):
hw/acpi: Don't include auxiliary vcpus in ACPI tables
softmmu: Add pause_all_vcpus_except_aux
migration: Stop non-aux vcpus before copying the last pages
migration: Call migration handler cleanup routines
docs/confidential-guest-live-migration.rst | 142 ++++++++++++
docs/confidential-guest-support.txt | 5 +
docs/index.rst | 1 +
qapi/migration.json | 26 +++
include/exec/ram_addr.h | 197 ++++++++++++++++
include/exec/ramblock.h | 3 +
include/exec/ramlist.h | 3 +-
include/hw/boards.h | 3 +
include/hw/core/cpu.h | 2 +
include/hw/i386/x86.h | 2 +-
include/sysemu/cpus.h | 2 +
linux-headers/linux/kvm.h | 13 ++
migration/confidential-ram.h | 23 ++
accel/kvm/kvm-all.c | 43 ++++
hw/acpi/cpu.c | 10 +
hw/core/cpu.c | 1 +
hw/core/machine.c | 7 +
hw/i386/acpi-build.c | 5 +
hw/i386/acpi-common.c | 5 +
hw/i386/pc.c | 7 +
hw/i386/x86.c | 10 +-
hw/isa/lpc_ich9.c | 3 +-
migration/confidential-ram.c | 258 +++++++++++++++++++++
migration/migration.c | 18 +-
migration/ram.c | 135 ++++++++++-
migration/savevm.c | 13 +-
softmmu/cpus.c | 68 +++++-
softmmu/runstate.c | 1 +
softmmu/vl.c | 3 +
target/i386/machine.c | 9 +
target/i386/sev.c | 25 +-
migration/meson.build | 6 +-
migration/trace-events | 4 +
33 files changed, 1027 insertions(+), 26 deletions(-)
create mode 100644 docs/confidential-guest-live-migration.rst
create mode 100644 migration/confidential-ram.h
create mode 100644 migration/confidential-ram.c
base-commit: 00d8ba9e0d62ea1c7459c25aeabf9c8bb7659462
--
2.20.1
- [RFC PATCH 00/26] Confidential guest live migration,
Dov Murik <=
- [RFC PATCH 01/26] linux-headers: Add definitions of KVM page encryption bitmap ioctls, Dov Murik, 2021/03/02
- [RFC PATCH 02/26] kvm: add support to sync the page encryption state bitmap, Dov Murik, 2021/03/02
- [RFC PATCH 12/26] migration: Add helpers to save confidential RAM, Dov Murik, 2021/03/02
- [RFC PATCH 04/26] hw/boards: Add aux flag to CPUArchId, Dov Murik, 2021/03/02
- [RFC PATCH 11/26] softmmu: Add pause_all_vcpus_except_aux, Dov Murik, 2021/03/02
- [RFC PATCH 16/26] migration: Load confidential guest RAM using migration helper, Dov Murik, 2021/03/02
- [RFC PATCH 06/26] hw/acpi: Don't include auxiliary vcpus in ACPI tables, Dov Murik, 2021/03/02
- [RFC PATCH 13/26] migration: Add helpers to load confidential RAM, Dov Murik, 2021/03/02
- [RFC PATCH 07/26] cpu: Add boolean aux field to CPUState, Dov Murik, 2021/03/02
- [RFC PATCH 23/26] target/i386: Re-sync kvm-clock after confidential guest migration, Dov Murik, 2021/03/02