Re: [PATCH v11 05/11] vfio/migration: Block multiple devices migration

[Alex Williamson]

On Tue, 16 May 2023 13:57:22 +0000
Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote:

> > -----Original Message-----
> > From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> > Sent: 16 May 2023 13:00
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: Avihai Horon <avihaih@nvidia.com>; qemu-devel@nongnu.org; Alex
> > Williamson <alex.williamson@redhat.com>; Juan Quintela
> > <quintela@redhat.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>;
> > Michael S. Tsirkin <mst@redhat.com>; Cornelia Huck <cohuck@redhat.com>;
> > Paolo Bonzini <pbonzini@redhat.com>; Vladimir Sementsov-Ogievskiy
> > <vsementsov@yandex-team.ru>; Cédric Le Goater <clg@redhat.com>; Yishai
> > Hadas <yishaih@nvidia.com>; Maor Gottlieb <maorg@nvidia.com>; Kirti
> > Wankhede <kwankhede@nvidia.com>; Tarun Gupta <targupta@nvidia.com>;
> > Joao Martins <joao.m.martins@oracle.com>
> > Subject: Re: [PATCH v11 05/11] vfio/migration: Block multiple devices
> > migration
> > 
> > On Tue, May 16, 2023 at 10:03:54AM +0000, Shameerali Kolothum Thodi
> > wrote:
> >   
> > > > Currently VFIO migration doesn't implement some kind of intermediate
> > > > quiescent state in which P2P DMAs are quiesced before stopping or
> > > > running the device. This can cause problems in multi-device migration
> > > > where the devices are doing P2P DMAs, since the devices are not stopped
> > > > together at the same time.
> > > >
> > > > Until such support is added, block migration of multiple devices.  
> > >
> > > Missed this one. Currently this blocks even if the attached devices are 
> > > not
> > > capable of P2P DMAs. eg; HiSilicon ACC devices. These are integrated end  
> > point  
> > > devices without any P2P capability between them. Is it Ok to check for
> > > VFIO_MIGRATION_P2P flag and allow if the devices are not supporting  
> > that?
> > 
> > Lacking VFIO_MIGRATION_P2P doesn't mean the device is incapable of
> > P2P, it means the migration can't support P2P.
> > 
> > We'd need some kind of new flag to check and such devices should be
> > blocked from creating P2P mappings. Basically we don't currently
> > fully support devices that are incapable of P2P operations.  
> 
> Ok. I will take a look.
> 
> > What happens on your platform if a guest tries to do P2P? Does the
> > platform crash?  
> 
> I am not sure. Since the devices are behind SMMU, I was under the assumption
> that we do have the guarantee of isolation here(grouping). Or this is 
> something
> we are worried only during migration?

Grouping doesn't guarantee that mappings cannot be created through the
SMMU between devices.  When we refer to devices being isolated between
groups, that only excludes internal P2P between devices, for example
across the internal link between functions with implementation specific
routing.  For group isolation, the guarantee is that DMA is always
routed upstream, not that the ultimate target cannot be another device.
To guarantee lack of P2P the SMMU would need to reject non-memory
translation targets.  Thanks,

Alex