Re: [PATCH] migration: support file: uri for source migration

[Nikolay Borisov]

On 12.09.22 г. 18:41 ч., Daniel P. Berrangé wrote:
> On Thu, Sep 08, 2022 at 01:26:32PM +0300, Nikolay Borisov wrote:
> > This is a prototype of supporting a 'file:' based uri protocol for
> > writing out the migration stream of qemu. Currently the code always
> > opens the file in DIO mode and adheres to an alignment of 64k to be
> > generic enough. However this comes with a problem - it requires copying
> > all data that we are writing (qemu metadata + guest ram pages) to a
> > bounce buffer so that we adhere to this alignment.
>
> The adhoc device metadata clearly needs bounce buffers since it
> is splattered all over RAM with no concern of alignemnt. THe use
> of bounce buffers for this shouldn't be a performance issue though
> as metadata is small relative to the size of the snapshot as a whole.

Bounce buffers can be eliminated altogether so long as we simply switch 
between buffered/DIO mode via fcntl.

> The guest RAM pages should not need bounce buffers at all when using
> huge pages, as alignment will already be way larger than we required.
> Guests with huge pages are the ones which are likely to have huge
> RAM sizes and thus need the DIO mode, so we should be sorted for that.
>
> When using small pages for guest RAM, if it is not already allocated
> with suitable alignment, I feel like we should be able to make it
> so that we allocate the RAM block with good alignemnt to avoid the
> need for bounce buffers. This would address the less common case of
> a guest with huge RAM size but not huge pages.

Ram blocks are generally allocated with good alignment due to them being 
mmaped(), however as I was toying with eliminating bounce buffers for 
ram I hit an issue where the page headers being written (8 bytes each) 
aren't aligned (naturally). Imo I think the on-disk format can be 
changed the following way:


<ramblock header, containing base address of ramblock>, each subsequent 
page is then written at an offset from the base address of the ramblock, 
that is it's index would be :

page_offset = page_addr - ramblock_base, Then the page is written at 
ramblock_base (in the file) + page_offset. This would eliminate the page 
headers altogether. This leaves aligning the initial ramblock header 
initially. However, this would lead to us potentially having to issue 1 
lseek per page to write - to adjust the the file position, which might 
not be a problem in itself but who knows. How dooes that sound to you?

> Thus if we assume guest RAM is suitably aligned, then we can avoid
> bounce buffers for RAM pages, while still using bounce buffers for
> the metadata.
>
> >                                                     With this code I get
> > the following performance results:
> >
> >        DIO              exec: cat > file         virsh --bypass-cache
> >        82                               77                                      
> >                 81
> >        82                           78                                          
> >         80
> >        80                           80                                          
> >         82
> >        82                           82                                          
> >         77
> >        77                           79                                          
> >         77
> >
> > AVG:  80.6                              79.2                                    
> >         79.4
> > stddev: 1.959                   1.720                                           
> > 2.05
> >
> > All numbers are in seconds.
> >
> > Those results are somewhat surprising to me as I'd expected doing the
> > writeout directly within qemu and avoiding copying between qemu and
> > virsh's iohelper process would result in a speed up. Clearly that's not
> > the case, I attribute this to the fact that all memory pages have to be
> > copied into the bounce buffer. There are more measurements/profiling
> > work that I'd have to do in order to (dis)prove this hypotheses and will
> > report back when I have the data.
>
> When using the libvirt iohelper we have mutliple CPUs involved. IOW the
> bounce buffer copy is taking place on a separate CPU from the QEMU
> migration loop. This ability to use multiple CPUs may well have balanced
> out any benefit from doing DIO on the QEMU side.
>
> If you eliminate bounce buffers for guest RAM and write it directly to
> the fixed location on disk, then we should see the benefit - and if not
> then something is really wrong in our thoughts.
>
> > However I'm sending the code now as I'd like to facilitate a discussion
> > as to whether this is an approach that would be acceptable to upstream
> > merging. Any ideas/comments would be much appreciated.
>
> AFAICT this impl is still using the existing on-disk format, where RAM
> pages are just written inline to the stream. For DIO benefit to be
> maximised we need the on-disk format to be changed, so that the guest
> RAM regions can be directly associated with fixed locations on disk.
> This also means that if guest dirties RAM while its saving, then we
> overwrite the existing content on disk, such that restore only ever
> needs to restore each RAM page once, instead of restoring every
> dirtied version.
>
>
> With regards,
> Daniel