Re: [PATCH] migration: support file: uri for source migration

[Daniel P . Berrangé]

On Mon, Sep 12, 2022 at 07:30:50PM +0300, Nikolay Borisov wrote:
> 
> 
> On 12.09.22 г. 18:41 ч., Daniel P. Berrangé wrote:
> > On Thu, Sep 08, 2022 at 01:26:32PM +0300, Nikolay Borisov wrote:
> > > This is a prototype of supporting a 'file:' based uri protocol for
> > > writing out the migration stream of qemu. Currently the code always
> > > opens the file in DIO mode and adheres to an alignment of 64k to be
> > > generic enough. However this comes with a problem - it requires copying
> > > all data that we are writing (qemu metadata + guest ram pages) to a
> > > bounce buffer so that we adhere to this alignment.
> > 
> > The adhoc device metadata clearly needs bounce buffers since it
> > is splattered all over RAM with no concern of alignemnt. THe use
> > of bounce buffers for this shouldn't be a performance issue though
> > as metadata is small relative to the size of the snapshot as a whole.
> 
> Bounce buffers can be eliminated altogether so long as we simply switch
> between buffered/DIO mode via fcntl.
> 
> > 
> > The guest RAM pages should not need bounce buffers at all when using
> > huge pages, as alignment will already be way larger than we required.
> > Guests with huge pages are the ones which are likely to have huge
> > RAM sizes and thus need the DIO mode, so we should be sorted for that.
> > 
> > When using small pages for guest RAM, if it is not already allocated
> > with suitable alignment, I feel like we should be able to make it
> > so that we allocate the RAM block with good alignemnt to avoid the
> > need for bounce buffers. This would address the less common case of
> > a guest with huge RAM size but not huge pages.
> 
> Ram blocks are generally allocated with good alignment due to them being
> mmaped(), however as I was toying with eliminating bounce buffers for ram I
> hit an issue where the page headers being written (8 bytes each) aren't
> aligned (naturally). Imo I think the on-disk format can be changed the
> following way:
> 
> 
> <ramblock header, containing base address of ramblock>, each subsequent page
> is then written at an offset from the base address of the ramblock, that is
> it's index would be :
> 
> page_offset = page_addr - ramblock_base, Then the page is written at
> ramblock_base (in the file) + page_offset. This would eliminate the page
> headers altogether. This leaves aligning the initial ramblock header
> initially. However, this would lead to us potentially having to issue 1
> lseek per page to write - to adjust the the file position, which might not
> be a problem in itself but who knows. How dooes that sound to you?

Yes, definitely. We don't want the headers on front of each page,
just one single large block. Looking forward to multi-fd, we don't
want to be using lseek at all, because that changes the file offset
for all threads using the FD. Instead we need to be able to use
pread/pwrite for writing the RAM blocks.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|