[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PULL 10/16] hw/vfio/pci-quirks: Support alternate offset for GPUDirect
From: |
Cédric Le Goater |
Subject: |
[PULL 10/16] hw/vfio/pci-quirks: Support alternate offset for GPUDirect Cliques |
Date: |
Fri, 30 Jun 2023 07:22:29 +0200 |
From: Alex Williamson <alex.williamson@redhat.com>
NVIDIA Turing and newer GPUs implement the MSI-X capability at the offset
previously reserved for use by hypervisors to implement the GPUDirect
Cliques capability. A revised specification provides an alternate
location. Add a config space walk to the quirk to check for conflicts,
allowing us to fall back to the new location or generate an error at the
quirk setup rather than when the real conflicting capability is added
should there be no available location.
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Cédric Le Goater <clg@redhat.com>
---
hw/vfio/pci-quirks.c | 41 ++++++++++++++++++++++++++++++++++++++++-
1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c
index f0147a050aaa..0ed2fcd53152 100644
--- a/hw/vfio/pci-quirks.c
+++ b/hw/vfio/pci-quirks.c
@@ -1490,6 +1490,9 @@ void vfio_setup_resetfn_quirk(VFIOPCIDevice *vdev)
* +---------------------------------+---------------------------------+
*
* https://lists.gnu.org/archive/html/qemu-devel/2017-08/pdfUda5iEpgOS.pdf
+ *
+ * Specification for Turning and later GPU architectures:
+ * https://lists.gnu.org/archive/html/qemu-devel/2023-06/pdf142OR4O4c2.pdf
*/
static void get_nv_gpudirect_clique_id(Object *obj, Visitor *v,
const char *name, void *opaque,
@@ -1530,7 +1533,9 @@ const PropertyInfo qdev_prop_nv_gpudirect_clique = {
static int vfio_add_nv_gpudirect_cap(VFIOPCIDevice *vdev, Error **errp)
{
PCIDevice *pdev = &vdev->pdev;
- int ret, pos = 0xC8;
+ int ret, pos;
+ bool c8_conflict = false, d4_conflict = false;
+ uint8_t tmp;
if (vdev->nv_gpudirect_clique == 0xFF) {
return 0;
@@ -1547,6 +1552,40 @@ static int vfio_add_nv_gpudirect_cap(VFIOPCIDevice
*vdev, Error **errp)
return -EINVAL;
}
+ /*
+ * Per the updated specification above, it's recommended to use offset
+ * D4h for Turing and later GPU architectures due to a conflict of the
+ * MSI-X capability at C8h. We don't know how to determine the GPU
+ * architecture, instead we walk the capability chain to mark conflicts
+ * and choose one or error based on the result.
+ *
+ * NB. Cap list head in pdev->config is already cleared, read from device.
+ */
+ ret = pread(vdev->vbasedev.fd, &tmp, 1,
+ vdev->config_offset + PCI_CAPABILITY_LIST);
+ if (ret != 1 || !tmp) {
+ error_setg(errp, "NVIDIA GPUDirect Clique ID: error getting cap list");
+ return -EINVAL;
+ }
+
+ do {
+ if (tmp == 0xC8) {
+ c8_conflict = true;
+ } else if (tmp == 0xD4) {
+ d4_conflict = true;
+ }
+ tmp = pdev->config[tmp + PCI_CAP_LIST_NEXT];
+ } while (tmp);
+
+ if (!c8_conflict) {
+ pos = 0xC8;
+ } else if (!d4_conflict) {
+ pos = 0xD4;
+ } else {
+ error_setg(errp, "NVIDIA GPUDirect Clique ID: invalid config space");
+ return -EINVAL;
+ }
+
ret = pci_add_capability(pdev, PCI_CAP_ID_VNDR, pos, 8, errp);
if (ret < 0) {
error_prepend(errp, "Failed to add NVIDIA GPUDirect cap: ");
--
2.41.0
- [PULL 00/16] vfio queue, Cédric Le Goater, 2023/06/30
- [PULL 03/16] migration: Enable switchover ack capability, Cédric Le Goater, 2023/06/30
- [PULL 01/16] migration: Add switchover ack capability, Cédric Le Goater, 2023/06/30
- [PULL 02/16] migration: Implement switchover ack logic, Cédric Le Goater, 2023/06/30
- [PULL 05/16] vfio/migration: Refactor vfio_save_block() to return saved data size, Cédric Le Goater, 2023/06/30
- [PULL 06/16] vfio/migration: Store VFIO migration flags in VFIOMigration, Cédric Le Goater, 2023/06/30
- [PULL 04/16] tests: Add migration switchover ack capability test, Cédric Le Goater, 2023/06/30
- [PULL 07/16] vfio/migration: Add VFIO migration pre-copy support, Cédric Le Goater, 2023/06/30
- [PULL 08/16] vfio/migration: Add support for switchover ack capability, Cédric Le Goater, 2023/06/30
- [PULL 14/16] MAINTAINERS: Promote Cédric to VFIO co-maintainer, Cédric Le Goater, 2023/06/30
- [PULL 10/16] hw/vfio/pci-quirks: Support alternate offset for GPUDirect Cliques,
Cédric Le Goater <=
- [PULL 11/16] vfio/pci: Call vfio_prepare_kvm_msi_virq_batch() in MSI retry path, Cédric Le Goater, 2023/06/30
- [PULL 09/16] vfio: Implement a common device info helper, Cédric Le Goater, 2023/06/30
- [PULL 12/16] vfio/migration: Reset bytes_transferred properly, Cédric Le Goater, 2023/06/30
- [PULL 13/16] vfio/migration: Make VFIO migration non-experimental, Cédric Le Goater, 2023/06/30
- [PULL 15/16] vfio/pci: Fix a segfault in vfio_realize, Cédric Le Goater, 2023/06/30
- [PULL 16/16] vfio/pci: Free leaked timer in vfio_realize error path, Cédric Le Goater, 2023/06/30
- Re: [PULL 00/16] vfio queue, Richard Henderson, 2023/06/30