[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-ppc] [PULL 29/44] spapr/xive: introduce a VM state change handler
From: |
David Gibson |
Subject: |
[Qemu-ppc] [PULL 29/44] spapr/xive: introduce a VM state change handler |
Date: |
Wed, 29 May 2019 16:50:02 +1000 |
From: Cédric Le Goater <address@hidden>
This handler is in charge of stabilizing the flow of event notifications
in the XIVE controller before migrating a guest. This is a requirement
before transferring the guest EQ pages to a destination.
When the VM is stopped, the handler sets the source PQs to PENDING to
stop the flow of events and to possibly catch a triggered interrupt
occuring while the VM is stopped. Their previous state is saved. The
XIVE controller is then synced through KVM to flush any in-flight
event notification and to stabilize the EQs. At this stage, the EQ
pages are marked dirty to make sure the EQ pages are transferred if a
migration sequence is in progress.
The previous configuration of the sources is restored when the VM
resumes, after a migration or a stop. If an interrupt was queued while
the VM was stopped, the handler simply generates the missing trigger.
Signed-off-by: Cédric Le Goater <address@hidden>
Reviewed-by: David Gibson <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: David Gibson <address@hidden>
---
hw/intc/spapr_xive_kvm.c | 96 ++++++++++++++++++++++++++++++++++++-
include/hw/ppc/spapr_xive.h | 1 +
2 files changed, 96 insertions(+), 1 deletion(-)
diff --git a/hw/intc/spapr_xive_kvm.c b/hw/intc/spapr_xive_kvm.c
index 8dd4f96e0b..735577a6f8 100644
--- a/hw/intc/spapr_xive_kvm.c
+++ b/hw/intc/spapr_xive_kvm.c
@@ -433,9 +433,100 @@ static void kvmppc_xive_get_queues(SpaprXive *xive, Error
**errp)
}
}
+/*
+ * The primary goal of the XIVE VM change handler is to mark the EQ
+ * pages dirty when all XIVE event notifications have stopped.
+ *
+ * Whenever the VM is stopped, the VM change handler sets the source
+ * PQs to PENDING to stop the flow of events and to possibly catch a
+ * triggered interrupt occuring while the VM is stopped. The previous
+ * state is saved in anticipation of a migration. The XIVE controller
+ * is then synced through KVM to flush any in-flight event
+ * notification and stabilize the EQs.
+ *
+ * At this stage, we can mark the EQ page dirty and let a migration
+ * sequence transfer the EQ pages to the destination, which is done
+ * just after the stop state.
+ *
+ * The previous configuration of the sources is restored when the VM
+ * runs again. If an interrupt was queued while the VM was stopped,
+ * simply generate a trigger.
+ */
+static void kvmppc_xive_change_state_handler(void *opaque, int running,
+ RunState state)
+{
+ SpaprXive *xive = opaque;
+ XiveSource *xsrc = &xive->source;
+ Error *local_err = NULL;
+ int i;
+
+ /*
+ * Restore the sources to their initial state. This is called when
+ * the VM resumes after a stop or a migration.
+ */
+ if (running) {
+ for (i = 0; i < xsrc->nr_irqs; i++) {
+ uint8_t pq = xive_source_esb_get(xsrc, i);
+ uint8_t old_pq;
+
+ old_pq = xive_esb_read(xsrc, i, XIVE_ESB_SET_PQ_00 + (pq << 8));
+
+ /*
+ * An interrupt was queued while the VM was stopped,
+ * generate a trigger.
+ */
+ if (pq == XIVE_ESB_RESET && old_pq == XIVE_ESB_QUEUED) {
+ xive_esb_trigger(xsrc, i);
+ }
+ }
+
+ return;
+ }
+
+ /*
+ * Mask the sources, to stop the flow of event notifications, and
+ * save the PQs locally in the XiveSource object. The XiveSource
+ * state will be collected later on by its vmstate handler if a
+ * migration is in progress.
+ */
+ for (i = 0; i < xsrc->nr_irqs; i++) {
+ uint8_t pq = xive_esb_read(xsrc, i, XIVE_ESB_GET);
+
+ /*
+ * PQ is set to PENDING to possibly catch a triggered
+ * interrupt occuring while the VM is stopped (hotplug event
+ * for instance) .
+ */
+ if (pq != XIVE_ESB_OFF) {
+ pq = xive_esb_read(xsrc, i, XIVE_ESB_SET_PQ_10);
+ }
+ xive_source_esb_set(xsrc, i, pq);
+ }
+
+ /*
+ * Sync the XIVE controller in KVM, to flush in-flight event
+ * notification that should be enqueued in the EQs and mark the
+ * XIVE EQ pages dirty to collect all updates.
+ */
+ kvm_device_access(xive->fd, KVM_DEV_XIVE_GRP_CTRL,
+ KVM_DEV_XIVE_EQ_SYNC, NULL, true, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ return;
+ }
+}
+
void kvmppc_xive_synchronize_state(SpaprXive *xive, Error **errp)
{
- kvmppc_xive_source_get_state(&xive->source);
+ /*
+ * When the VM is stopped, the sources are masked and the previous
+ * state is saved in anticipation of a migration. We should not
+ * synchronize the source state in that case else we will override
+ * the saved state.
+ */
+ if (runstate_is_running()) {
+ kvmppc_xive_source_get_state(&xive->source);
+ }
/* EAT: there is no extra state to query from KVM */
@@ -515,6 +606,9 @@ void kvmppc_xive_connect(SpaprXive *xive, Error **errp)
"xive.tima", tima_len, xive->tm_mmap);
sysbus_init_mmio(SYS_BUS_DEVICE(xive), &xive->tm_mmio);
+ xive->change = qemu_add_vm_change_state_handler(
+ kvmppc_xive_change_state_handler, xive);
+
kvm_kernel_irqchip = true;
kvm_msi_via_irqfd_allowed = true;
kvm_gsi_direct_mapping = true;
diff --git a/include/hw/ppc/spapr_xive.h b/include/hw/ppc/spapr_xive.h
index 7e49badd8c..734662c12a 100644
--- a/include/hw/ppc/spapr_xive.h
+++ b/include/hw/ppc/spapr_xive.h
@@ -42,6 +42,7 @@ typedef struct SpaprXive {
/* KVM support */
int fd;
void *tm_mmap;
+ VMChangeStateEntry *change;
} SpaprXive;
/*
--
2.21.0
- [Qemu-ppc] [PULL 16/44] spapr/xive: print out the EQ page address in the monitor, (continued)
- [Qemu-ppc] [PULL 16/44] spapr/xive: print out the EQ page address in the monitor, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 26/44] spapr/xive: add KVM support, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 17/44] Fix typo on "info pic" monitor cmd output for xive, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 24/44] spapr: Fix phb_placement backwards compatibility, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 20/44] spapr/xive: Sanity checks of OV5 during CAS, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 28/44] spapr/xive: add state synchronization with KVM, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 18/44] target/ppc: Optimise VSX_LOAD_SCALAR_DS and VSX_VECTOR_LOAD_STORE, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 25/44] spapr: Print out extra hints when CAS negotiation of interrupt mode fails, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 23/44] target/ppc: Use vector variable shifts for VSL, VSR, VSRA, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 34/44] spapr: check for the activation of the KVM IRQ device, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 29/44] spapr/xive: introduce a VM state change handler,
David Gibson <=
- [Qemu-ppc] [PULL 19/44] target/ppc: Fix xvabs[sd]p, xvnabs[sd]p, xvneg[sd]p, xvcpsgn[sd]p, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 22/44] spapr: Add forgotten capability to migration stream, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 31/44] spapr/xive: activate KVM support, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 27/44] spapr/xive: add hcall support when under KVM, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 21/44] target/ppc: Set PSSCR_EC on cpu halt to prevent spurious wakeup, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 32/44] sysbus: add a sysbus_mmio_unmap() helper, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 44/44] ppc/pnv: add dummy XSCOM registers for PRD initialization, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 43/44] ppc/pnv: introduce new skiboot platform properties, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 36/44] spapr/irq: initialize the IRQ device only once, David Gibson, 2019/05/29
- [Qemu-ppc] [PULL 41/44] spapr: change default interrupt mode to 'dual', David Gibson, 2019/05/29