Re: "Instant clone" with Qemu?

[Philipp Hahn]

Hello Stefan,

Am 15.12.23 um 16:21 schrieb Stefan Hajnoczi:
> > Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:
> > > On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
> >   >
> > > > by accident I stumbled over "VMware Instant Clone" ¹, which allows
> > > > cloning of running VMs by copy-on-write-sharing the disk images and
> > > > memory content; the network MAC address gets changed (or a different
> > > > bridge is used?).
> > > > I wonder if something similar can also be done with Qemu? My current
> > > > solution would be to:
> > > > - start and install the VM
> > > > - create a live-snapshot into the qcow2 file
> > > > - clone the disk image, e.g. put a qcow2 overlay on it per clone
> > > > - start and restore the clones from that live-snapshot
> > > > - put the clones in individual bridges and let the host do some network
> > > > address translation (NAT) to give each clone a unique external IP address.
> > > >
> > > > Has someone done something similar or is there even a better alternative?
> > > >
> > > > Background: our test suite currently provisions a set of multiple VMs,
> > > > which are dependent on each other. Provisioning them takes sometimes
> > > > many hours. After that the test suite runs inside these VMs and again
> > > > takes many hours.
> > > > I'd like to speed that up by parallelizing these tests, e.g.
> > > > 1. setup the VM environment once
> > > > 2. clone the VM environments as the resources allow
> > > > 3. distribute tests over these environments to run in parallel and to
> > > > allow running flaky tests multiple times from a clean clone again
> > >
> > > It would be simplest to use qcow2 backing files and boot each new
> > > instance from scratch. This involves setting up a master image and
> > > then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
> > > the instance image. You may be able to use systemd or your distro's
> > > "first boot" functionality to recreate unique IDs and cryptographic
> > > keys when the new instance boots.
> >
> > Actually I do not want to modify the clones at all: While the machine ID
> > is probably less interesting to others, I can even live with re-using
> > the SSH keys as this is only for *internal* testing: I can tell `ssh` to
> > not check the keys as I can control all the networking, so security is
> > of little concern here.
> >
> > > If you really want to use a RAM snapshot then I suggest creating a
> > > qcow2 master image with the savevm command and using "cp
> > > --reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
> > > of the qcow2 file. You'll need some custom scripts to recreate unique
> > > IDs and cryptographic keys inside the new instance after loadvm.
> >
> > Is there a major difference between doing a "savevm" to an external file
> > and doing a live snapshot, which stores the "savevm" inside the qcow2
> > file itself. The later has the benefit for me, that I only have to
> > handle one file; I could even store it for later use if needed.
>
> With the reflink approach you still snapshot the VM into the original
> qcow2 file (with the "savevm" command), not into an external file. The
> reflink creates an efficient copy of the file for each instance of the
> VM that you with to clone. But since you've said you don't want to
> modify the clones at all, maybe this approach is overkill because you
> have to manage these new qcow2 files.
>
> Have you tried the -snapshot command-line option?

Thank you for the reminder, I already used it in other contexts but 
forgot it here. I'll add it to my toolshed.

Regarding reflink: that requires FS support, which AFAIKS is only 
supported by BTRFS and XFS (and OCFS2). So using `-snapshot` or an 
explicit `qemu-img create -f qcow2 -b $base.qcow2 $overlay.qcow2` works 
on any FS.

> > My main problem currently is cloning the MAC address: As our product is
> > an operating system the MAC addresses of the involved systems is stored
> > in some databases; while in most cases they are not required, I do not
> > want to hunt for these in all kind of different locations and change
> > them to some cloned MAC address.
> > I already had a look at "Virtual Routing and Forwarding"², which allows
> > me to resue the same MAC addresses in different network bridge
> > interfaces, but what I did not yet get to work is the "routing" between
> > them. I found some very nice articles³⁴ on how to do NAT with VRF, but
> > it is not yet working.
>
> I'm not knowledgeable about VRF. You could also use -netdev
> user,hostfwd=tcp::$VM_SSH_NAT_PORT-:22 where VM_SSH_NAT_PORT is a
> unique port assigned by the script that launches the guest. That way
> each guest can have the same MAC address and IP address but receive
> incoming SSH connections.

Good to know, thank you. My problem is that I have to clone multiple VMs 
belonging together, they must be able to communicate with each other 
using their unmodified IP addresses; I only need to connect to one of 
them from the outside; something like a "jump host".

Thanks again
Philipp