[RFC] vfio/migration: The way we start dirty tracking for the VFIO container?

[Zenghui Yu]

Hi folks,

I'm looking at the way we start dirty tracking for the vfio container
(we start it at the save_setup stage of a device) and have a few
questions about it.  Please fix me up if I had missed something obvious.


- We may end up with the situation where we try to get dirty bitmap of
  the vfio container whilst the dirty tracking hasn't yet been enabled.

    ram_save_setup
        ram_init_bitmaps
            migration_bitmap_sync_precopy (i.e., vfio_listener_log_sync)
    vfio_save_setup
        vfio_migration_set_state += _SAVING
        vfio_set_dirty_page_tracking(true)

  Practically there is no problem since in vfio_listener_log_sync(), we
  will check that if all devices are having the _SAVING state *before*
  actually syncing dirty bitmap.  As we add _SAVING to the device state
  exactly before starting dirty tracking, we'll actually do nothing in
  the first run of vfio_listener_log_sync().

  But I'm wondering what prevents us from starting dirty tracking via
  the "standard" log_start() callback of the vfio container.


- Before start dirty tracking, we will check and ensure that the device
  is at _SAVING state and return error otherwise.  But the question is
  that what is the rationale?  Why does the VFIO_IOMMU_DIRTY_PAGES ioctl
  have something to do with the device state?

  VFIO_IOMMU_DIRTY_PAGES is clearly designed as a API for type1 VFIO
  IOMMU.  Is it possible that the *device* itself becomes the actual
  dirty bitmap provider of this ioctl (e.g., the device is smart enough
  to track its own dirty pages during migration)?  This is the only case
  I can think out where we may have to set the device to pre-copy state
  before starting dirty tracking, so that device can report dirty pages
  properly in response to the GET_BITMAP ioctl.  But afaict it isn't
  covered by the existing documentation [1][2]...

  Btw, it isn't clear that what possible dirty bitmap providers will be
  (IOMMU, devices, etc) behind the back of the VFIO_IOMMU_DIRTY_PAGES
  ioctl.  Although userspace can be unaware of any of this, it'd be
  appreciated if someone can clarify it.


- If there is indeed some dependency between the device state and the
  VFIO_IOMMU_DIRTY_PAGES_FLAG_START operation, shouldn't we check that
  *all* devices are now having the _SAVING state before START, rather
  than just checking the *given* device in its (per-device) save_setup
  stage?

[1] https://git.kernel.org/torvalds/c/a8a24f3f6e38
[2] https://git.kernel.org/torvalds/c/b704fd14a06f


Thanks,
Zenghui