[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [RFC PATCH v2 00/13] SMMUv3 nested translation support
From: |
Eric Auger |
Subject: |
Re: [RFC PATCH v2 00/13] SMMUv3 nested translation support |
Date: |
Thu, 18 Apr 2024 20:11:06 +0200 |
User-agent: |
Mozilla Thunderbird |
Hi Mostafa,
On 4/8/24 16:08, Mostafa Saleh wrote:
> Currently, QEMU supports emulating either stage-1 or stage-2 SMMUs
> but not nested instances.
> This patch series adds support for nested translation in SMMUv3,
> this is controlled by property “arm-smmuv3.stage=nested”, and
> advertised to guests as (IDR0.S1P == 1 && IDR0.S2P == 2)
>
> Main changes(architecture):
> ============================
> 1) CDs are considered IPA and translated with stage-2.
> 2) TTBx and tables for stage-1 are considered IPA and translated
> with stage-2.
> 3) Translate the IPA address with stage-2.
>
> TLBs:
> ======
> TLBs are the most tricky part.
>
> 1) General design
> Unified(Combined) design is used, where entries with ASID=-1 are
> IPAs(cached from stage-2 config)
>
> TLBs are also modified to cache 2 permissions, a new permission added
> "parent_perm."
>
> For non-nested configuration, perm == parent_perm and nothing
> changes. This is used to know which stage to use in case there is
> a permission fault from a TLB entry.
>
> 2) Caching in TLB
> Stage-1 and stage-2 are inserted in the TLB as is.
> For nested translation, both entries are combined into one TLB
> entry. The size (level and granule) are chosen from the smallest entries.
> That means that a stage-1 translation can be cached with sage-2
> granule in key, this is take into account lookup.
is that a correct understanding that with the current implementation, in
nested mode, you end up with combined S1 + S2 entries (IOVA -> PA) and
S2 entries (IPA -> PA)?
Out of cusiosity, how did you end up with that choice? Have you made
some perf assessment compared to separate S1 and S2 entries? I guess it
is a complex topic and choice.
Thanks
Eric
>
> 3) TLB Lookup
> TLB lookup already uses ASID in key, so it can distinguish between
> stage-1 and stage-2.
> And as mentioned above, the granule for stage-1 can be different,
> If stage-1 lookup failed, we try again with the stage-2 granule.
>
> 4) TLB invalidation
> - Address invalidation is split, for IOVA(CMD_TLBI_NH_VA
> /CMD_TLBI_NH_VAA) and IPA(CMD_TLBI_S2_IPA) based on ASID value
> - CMD_TLBI_NH_ASID/CMD_TLBI_NH_ALL: Consider VMID if stage-2 is
> supported, and invalidate stage-1 only by VMIDs
>
> As far as I understand, this is compliant with the ARM architecture:
> - ARM ARM DDI 0487J.a: RLGSCG, RTVTYQ, RGNJPZ
> - ARM IHI 0070F.b: 16.2 Caching
>
> An alternative approach would be to instantiate 2 TLBs, one per each
> stage. I haven’t investigated that.
>
> Others
> =======
> - Advertise SMMUv3.2-S2FWB, it is NOP for QEMU as it doesn’t support
> attributes.
>
> - OAS: A typical setup with nesting is to share CPU stage-2 with the
> SMMU, and according to the user manual, SMMU OAS must match the
> system physical address.
>
> This was discussed before in
> https://lore.kernel.org/all/20230226220650.1480786-11-smostafa@google.com/
> The implementation here, follows the discussion, where migration is
> added and oas is set up from the board (virt). However, the OAS is
> chosen based on the CPU PARANGE as there is no fixed one.
>
> - For nested configuration, IOVA notifier only notifies for stage-1
> invalidations (as far as I understand this is the intended
> behaviour as it notifies for IOVA)
>
> - Stop ignoring VMID for stage-1 if stage-2 is also supported.
>
>
> Future improvements:
> =====================
> 1) One small improvement, that I don’t think it’s worth the extra
> complexity, is in case of Stage-1 TLB miss for nested translation,
> we can do stage-1 walk and lookup for stage-2 TLBs, instead of
> doing the full walk.
>
> 2) Patch 0006 (hw/arm/smmuv3: Translate CD and TT using stage-2 table)
> introduces a macro to use functions that rely on cfg for stage-2,
> I don’t like it. However, I didn’t find a simple way around it,
> either we change many functions to have a separate stage argument,
> or add another arg in config, which is probably more code.
>
> Testing
> ========
> 1) IOMMUFD + VFIO
> Kernel:
> https://lore.kernel.org/all/cover.1683688960.git.nicolinc@nvidia.com/
> VMM:
> https://qemu-devel.nongnu.narkive.com/o815DqpI/rfc-v5-0-8-arm-smmuv3-emulation-support
>
> By assigning
> “virtio-net-pci,netdev=net0,disable-legacy=on,iommu_platform=on,ats=on”,
> to a guest VM (on top of QEMU guest) with VIFO and IOMMUFD.
>
> 2) Work in progress prototype I am hacking on for nesting on KVM
> (this is nowhere near complete, and misses many stuff but it
> doesn't require VMs/VFIO) also with virtio-net-pci and git
> cloning a bunch of stuff and also observing traces.
>
> https://android-kvm.googlesource.com/linux/+log/refs/heads/smostafa/android15-6.6-smmu-nesting-wip
>
> I also modified the Linux driver to test with mixed granules/levels.
>
> hw/arm/smmuv3: Split smmuv3_translate() better viewed with --color-moved
>
> Changes in v2:
> v1:
> https://lore.kernel.org/qemu-devel/20240325101442.1306300-1-smostafa@google.com/
> - Collected Eric Rbs
> - Rework TLB to rely on VMID/ASID instead of an extra key.
> - Fixed TLB issue with large stage-1 reported by Julian.
> - Cap the OAS to 48 bits as PTW doesn’t support 52 bits.
> - Fix ASID/VMID representation in some contexts as 16 bits while
> they can be -1
> - Increase visibility in trace points
>
>
> Mostafa Saleh (13):
> hw/arm/smmu: Use enum for SMMU stage
> hw/arm/smmu: Split smmuv3_translate()
> hw/arm/smmu: Consolidate ASID and VMID types
> hw/arm/smmuv3: Translate CD and TT using stage-2 table
> hw/arm/smmu-common: Support nested translation
> hw/arm/smmu: Support nesting in smmuv3_range_inval()
> hw/arm/smmu: Support nesting in the rest of commands
> hw/arm/smmuv3: Support nested SMMUs in smmuv3_notify_iova()
> hw/arm/smmuv3: Support and advertise nesting
> hw/arm/smmuv3: Advertise S2FWB
> hw/arm/smmu: Refactor SMMU OAS
> hw/arm/smmuv3: Add property for OAS
> hw/arm/virt: Set SMMU OAS based on CPU PARANGE
>
> hw/arm/smmu-common.c | 306 +++++++++++++++++++++----
> hw/arm/smmuv3-internal.h | 16 +-
> hw/arm/smmuv3.c | 418 ++++++++++++++++++++++-------------
> hw/arm/trace-events | 18 +-
> hw/arm/virt.c | 14 +-
> include/hw/arm/smmu-common.h | 57 ++++-
> include/hw/arm/smmuv3.h | 1 +
> target/arm/cpu.h | 2 +
> target/arm/cpu64.c | 5 +
> 9 files changed, 608 insertions(+), 229 deletions(-)
>
- Re: [RFC PATCH v2 06/13] hw/arm/smmu: Support nesting in smmuv3_range_inval(), (continued)
- [RFC PATCH v2 09/13] hw/arm/smmuv3: Support and advertise nesting, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 07/13] hw/arm/smmu: Support nesting in the rest of commands, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 11/13] hw/arm/smmu: Refactor SMMU OAS, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 12/13] hw/arm/smmuv3: Add property for OAS, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 10/13] hw/arm/smmuv3: Advertise S2FWB, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 13/13] hw/arm/virt: Set SMMU OAS based on CPU PARANGE, Mostafa Saleh, 2024/04/08
- [RFC PATCH v2 08/13] hw/arm/smmuv3: Support nested SMMUs in smmuv3_notify_iova(), Mostafa Saleh, 2024/04/08
- Re: [RFC PATCH v2 00/13] SMMUv3 nested translation support,
Eric Auger <=