Re: bitmaps -- copying allocation status into dirty bitmaps

[John Snow]

On 11/4/19 6:27 AM, Max Reitz wrote:
> On 04.11.19 12:21, Max Reitz wrote:
>> On 01.11.19 16:42, John Snow wrote:
>>> Hi, in one of my infamously unreadable and long status emails, I
>>> mentioned possibly wanting to copy allocation data into bitmaps as a way
>>> to enable users to create (external) snapshots from outside of the
>>> libvirt/qemu context.
>>>
>>> (That is: to repair checkpoints in libvirt after a user extended the
>>> backing chain themselves, you want to restore bitmap information for
>>> that node. Conveniently, this information IS the allocation map, so we
>>> can do this.)
>>>
>>> It came up at KVM Forum that we probably do want this, because oVirt
>>> likes the idea of being able to manipulate these chains from outside of
>>> libvirt/qemu.
>>>
>>> Denis suggested that instead of a new command, we can create a special
>>> name -- maybe "#ALLOCATED" or something similar that can never be
>>> allocated as a user-defined bitmap name -- as a special source for the
>>> merge command.
>>>
>>> You'd issue a merge from "#ALLOCATED" to "myBitmap0" to copy the current
>>> allocation data into "myBitmap0", for instance.
>>
>> Sounds fun, but is there actually any use for this if the only purpose
>> is to work as a source for merge?
>>
>> I mean, it would be interesting if it worked exactly like a perma-RO
>> pseudo-bitmap that whenever you try to get data from it performs a
>> block-status call.  But as you say, that would probably be too slow, and
>> it would take a lot of code modifications, so I wonder if there is
>> actually any purpose for this.
>>
>>> Some thoughts:
>>>
>>> - The only commands where this pseudo-bitmap makes sense is merge.
>>> enable/disable/remove/clear/add don't make sense here.
>>>
>>> - This pseudo bitmap might make sense for backup, but it's not needed;
>>> you can just merge into an empty/enabled bitmap and then use that.
>>>
>>> - Creating an allocation bitmap on-the-fly is probably not possible
>>> directly in the merge command, because the disk status calls might take
>>> too long...
>>>
>>> Hm, actually, I'm not sure how to solve that one. Merge would need to
>>> become a job (or an async QMP command?) or we'd need to keep an
>>> allocation bitmap object around and in-sync. I don't really want to do
>>> either, so maybe I'm missing an obvious/better solution.
>>
>> All of what you wrote in this mail makes me think it would make much
>> more sense to just add a “block-dirty-bitmap-create-from” job with an
>> enum of targets.  (One of which would be “allocated-blocks”.)

Sounds good. (What are the other targets? Questions-for-later?)

> 
> I forgot to add that of course the advantage of a pseudo-bitmap would be
> that it’s always up to date, but as you said, it would be slow to query
> (and it might even yield, which isn’t what callers expect) and at least
> for block allocation, it seems unnecessary to me (because writes will
> keep the new bitmap created from allocated-blocks up-to-date).
> 
> Max
> 

Who knows what's happened in the month since I've been gone, but I think
I agree completely with your assessment.

In our meeting with Denis it seemed like it was the optimal thing to
make a pseudo-bitmap for merge so we didn't have to add a new command,
but I think it's clear that the async properties are going to prohibit
that nice solution and we will indeed need a job.

--js