Re: [PATCH v2 15/18] target/riscv: make riscv_isa_string_ext() KVM compatible

[Andrew Jones]

On Tue, Jun 20, 2023 at 07:05:18PM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 6/19/23 06:54, Andrew Jones wrote:
> > On Tue, Jun 13, 2023 at 05:58:54PM -0300, Daniel Henrique Barboza wrote:
> > > The isa_edata_arr[] is used by riscv_isa_string_ext() to create the
> > > riscv,isa DT attribute. isa_edata_arr[] is kept in sync with the TCG
> > > property vector riscv_cpu_extensions[], i.e. all TCG properties from
> > > this vector that has a riscv,isa representation are included in
> > > isa_edata_arr[].
> > > 
> > > KVM doesn't implement all TCG properties, but allow them to be created
> > > anyway to not force an API change between TCG and KVM guests. Some of
> > > these TCG-only extensions are defaulted to 'true', and users are also
> > > allowed to enable them. KVM doesn't care, but riscv_isa_string_ext()
> > > does. The result is that these TCG-only enabled extensions will appear
> > > in the riscv,isa DT string under KVM.
> > > 
> > > To avoid code repetition and re-use riscv_isa_string_ext() for KVM
> > > guests we'll make a couple of tweaks:
> > > 
> > > - set env->priv_ver to 'LATEST' for the KVM 'host' CPU. This is needed
> > >    because riscv_isa_string_ext() makes a priv check for each extension
> > >    before including them in the ISA string. KVM doesn't care about
> > >    env->priv_ver, since it's part of the TCG-only CPU validation, so this
> > >    change is benign for KVM;
> > > 
> > > - add a new 'kvm_available' flag in isa_ext_data struct. This flag is
> > >    set via a new ISA_EXT_DATA_ENTRY_KVM macro to report that, for a given
> > >    extension, KVM also supports it. riscv_isa_string_ext() then can check
> > >    if a given extension is known by KVM and skip it if it's not.
> > > 
> > > This will allow us to re-use riscv_isa_string_ext() for KVM guests.
> > > 
> > > Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
> > > ---
> > >   target/riscv/cpu.c | 28 ++++++++++++++++++++--------
> > >   1 file changed, 20 insertions(+), 8 deletions(-)
> > > 
> > > diff --git a/target/riscv/cpu.c b/target/riscv/cpu.c
> > > index a4f3ed0c17..a773c09645 100644
> > > --- a/target/riscv/cpu.c
> > > +++ b/target/riscv/cpu.c
> > > @@ -44,11 +44,15 @@ struct isa_ext_data {
> > >       const char *name;
> > >       int min_version;
> > >       int ext_enable_offset;
> > > +    bool kvm_available;
> > >   };
> > >   #define ISA_EXT_DATA_ENTRY(_name, _min_ver, _prop) \
> > >       {#_name, _min_ver, offsetof(struct RISCVCPUConfig, _prop)}
> > > +#define ISA_EXT_DATA_ENTRY_KVM(_name, _min_ver, _prop) \
> > > +    {#_name, _min_ver, offsetof(struct RISCVCPUConfig, _prop), true}
> > > +
> > >   /*
> > >    * Here are the ordering rules of extension naming defined by RISC-V
> > >    * specification :
> > > @@ -68,14 +72,17 @@ struct isa_ext_data {
> > >    *
> > >    * Single letter extensions are checked in 
> > > riscv_cpu_validate_misa_priv()
> > >    * instead.
> > > + *
> > > + * ISA_EXT_DATA_ENTRY_KVM() is used to indicate that the extension is
> > > + * also known by the KVM driver. If unsure, use ISA_EXT_DATA_ENTRY().
> > >    */
> > >   static const struct isa_ext_data isa_edata_arr[] = {
> > > -    ISA_EXT_DATA_ENTRY(zicbom, PRIV_VERSION_1_12_0, ext_icbom),
> > > -    ISA_EXT_DATA_ENTRY(zicboz, PRIV_VERSION_1_12_0, ext_icboz),
> > > +    ISA_EXT_DATA_ENTRY_KVM(zicbom, PRIV_VERSION_1_12_0, ext_icbom),
> > > +    ISA_EXT_DATA_ENTRY_KVM(zicboz, PRIV_VERSION_1_12_0, ext_icboz),
> > >       ISA_EXT_DATA_ENTRY(zicond, PRIV_VERSION_1_12_0, ext_zicond),
> > >       ISA_EXT_DATA_ENTRY(zicsr, PRIV_VERSION_1_10_0, ext_icsr),
> > >       ISA_EXT_DATA_ENTRY(zifencei, PRIV_VERSION_1_10_0, ext_ifencei),
> > > -    ISA_EXT_DATA_ENTRY(zihintpause, PRIV_VERSION_1_10_0, 
> > > ext_zihintpause),
> > > +    ISA_EXT_DATA_ENTRY_KVM(zihintpause, PRIV_VERSION_1_10_0, 
> > > ext_zihintpause),
> > >       ISA_EXT_DATA_ENTRY(zawrs, PRIV_VERSION_1_12_0, ext_zawrs),
> > >       ISA_EXT_DATA_ENTRY(zfh, PRIV_VERSION_1_11_0, ext_zfh),
> > >       ISA_EXT_DATA_ENTRY(zfhmin, PRIV_VERSION_1_11_0, ext_zfhmin),
> > > @@ -89,7 +96,7 @@ static const struct isa_ext_data isa_edata_arr[] = {
> > >       ISA_EXT_DATA_ENTRY(zcmp, PRIV_VERSION_1_12_0, ext_zcmp),
> > >       ISA_EXT_DATA_ENTRY(zcmt, PRIV_VERSION_1_12_0, ext_zcmt),
> > >       ISA_EXT_DATA_ENTRY(zba, PRIV_VERSION_1_12_0, ext_zba),
> > > -    ISA_EXT_DATA_ENTRY(zbb, PRIV_VERSION_1_12_0, ext_zbb),
> > > +    ISA_EXT_DATA_ENTRY_KVM(zbb, PRIV_VERSION_1_12_0, ext_zbb),
> > >       ISA_EXT_DATA_ENTRY(zbc, PRIV_VERSION_1_12_0, ext_zbc),
> > >       ISA_EXT_DATA_ENTRY(zbkb, PRIV_VERSION_1_12_0, ext_zbkb),
> > >       ISA_EXT_DATA_ENTRY(zbkc, PRIV_VERSION_1_12_0, ext_zbkc),
> > > @@ -114,13 +121,13 @@ static const struct isa_ext_data isa_edata_arr[] = {
> > >       ISA_EXT_DATA_ENTRY(zhinxmin, PRIV_VERSION_1_12_0, ext_zhinxmin),
> > >       ISA_EXT_DATA_ENTRY(smaia, PRIV_VERSION_1_12_0, ext_smaia),
> > >       ISA_EXT_DATA_ENTRY(smstateen, PRIV_VERSION_1_12_0, ext_smstateen),
> > > -    ISA_EXT_DATA_ENTRY(ssaia, PRIV_VERSION_1_12_0, ext_ssaia),
> > > +    ISA_EXT_DATA_ENTRY_KVM(ssaia, PRIV_VERSION_1_12_0, ext_ssaia),
> > >       ISA_EXT_DATA_ENTRY(sscofpmf, PRIV_VERSION_1_12_0, ext_sscofpmf),
> > > -    ISA_EXT_DATA_ENTRY(sstc, PRIV_VERSION_1_12_0, ext_sstc),
> > > +    ISA_EXT_DATA_ENTRY_KVM(sstc, PRIV_VERSION_1_12_0, ext_sstc),
> > >       ISA_EXT_DATA_ENTRY(svadu, PRIV_VERSION_1_12_0, ext_svadu),
> > > -    ISA_EXT_DATA_ENTRY(svinval, PRIV_VERSION_1_12_0, ext_svinval),
> > > +    ISA_EXT_DATA_ENTRY_KVM(svinval, PRIV_VERSION_1_12_0, ext_svinval),
> > >       ISA_EXT_DATA_ENTRY(svnapot, PRIV_VERSION_1_12_0, ext_svnapot),
> > > -    ISA_EXT_DATA_ENTRY(svpbmt, PRIV_VERSION_1_12_0, ext_svpbmt),
> > > +    ISA_EXT_DATA_ENTRY_KVM(svpbmt, PRIV_VERSION_1_12_0, ext_svpbmt),
> > 
> > This approach looks a bit difficult to maintain (it's hard to even see the
> > _KVM macro uses). The approach will be even more difficult if we add more
> > accelerators. I feel like we need an extension class where objects of that
> > class can be passed to functions like riscv_isa_string_ext(). And then we
> > also need tcg-extension and kvm-extension classes which inherit from
> > the extension class. We can then keep the lists of extensions separate, as
> > you originally proposed, as each list will be of its own type.
> 
> So, after coding around a little, I didn't manage to create a KVM list in
> kvm.c file because of how ARRAY_SIZE() (a macro that riscv_isa_string_ext())
> calculates the array size. If the list is exported from another file the macro
> fails to calculate the size of the array. Similar issues also happens when 
> trying
> to use kvm_multi_ext_cfgs[] in this function.
> 
> This means that both tcg and kvm arrays ended up in cpu.c.

Hmm, I'd really rather we don't have a kvm array in cpu.c. Going back to
duplicating functions like riscv_isa_string_ext() into kvm.c is better,
IMO.

> The tcg array is
> left untouched (isa_edata_arr). The KVM array uses the same type TCG already
> uses (isa_ext_edata) because it's good enough for KVM as well. I removed the
> 'kvm_available' flag since we're going to manually edit the array.
> 
> riscv_isa_string_ext() is then changed to switch between the tcg/kvm array
> as required. The rest of the logic of the function stood the same. I'll also
> add a pre-patch prior to all these changes to simplify riscv_isa_string_ext()
> a bit when running under TCG.

If we have to teach riscv_isa_string_ext() about both tcg and kvm cfg
types (and any other functions that come along), then this approach
isn't all that helpful anyway.

Thanks,
drew