[Qemu-discuss] domain crashes when blockcommit

[Lentes, Bernd]

Hi,

this is a continuation of 
https://www.redhat.com/archives/libvirt-users/2019-June/msg00003.html.
I have a HA-Cluster with 2 nodes on which several domains are running with 
libvirt and qemu.
Each night i create a snapshot of the domains, which are shutdown prior.
Snapshots are created with virsh snapshot-create.
The procedure is:
- shutdown domains
- create snapshots
- start domains
- copy backing raw file to a cifs server
- blockcommit snapshot via virsh blockcommit domain /path/to/snapshot --verbose 
--active --pivot

Mostly this works quite well.
But sometimes one domain crashes on blockcommit.

Here the excerpts from the syslog on the hosts:

> 2019-06-01T03:05:31.620725+02:00 ha-idg-2 systemd-coredump[14253]: Core 
> Dumping has been disabled for process 30590 (qemu-system-x86).
> 2019-06-01T03:05:31.712673+02:00 ha-idg-2 systemd-coredump[14253]: Process 
> 30590 (qemu-system-x86) of user 488 dumped core.
> 2019-06-01T03:05:32.173272+02:00 ha-idg-2 kernel: [294682.387828] br0: port 
> 4(vnet2) entered disabled state
> 2019-06-01T03:05:32.177111+02:00 ha-idg-2 kernel: [294682.388384] device 
> vnet2 left promiscuous mode
> 2019-06-01T03:05:32.177122+02:00 ha-idg-2 kernel: [294682.388391] br0: port 
> 4(vnet2) entered disabled state
> 2019-06-01T03:05:32.208916+02:00 ha-idg-2 wickedd[2954]: error retrieving tap 
> attribute from sysfs
> 2019-06-01T03:05:41.395685+02:00 ha-idg-2 systemd-machined[2824]: Machine 
> qemu-31-severin terminated.
> 
> 
> 2019-06-08T05:59:17.502899+02:00 ha-idg-1 systemd-coredump[31089]: Core 
> Dumping has been disabled for process 19489 (qemu-system-x86).
> 2019-06-08T05:59:17.523050+02:00 ha-idg-1 systemd-coredump[31089]: Process 
> 19489 (qemu-system-x86) of user 489 dumped core.
> 2019-06-08T05:59:17.650334+02:00 ha-idg-1 kernel: [999258.577132] br0: port 
> 9(vnet7) entered disabled state
> 2019-06-08T05:59:17.650354+02:00 ha-idg-1 kernel: [999258.578103] device 
> vnet7 left promiscuous mode
> 2019-06-08T05:59:17.650355+02:00 ha-idg-1 kernel: [999258.578108] br0: port 
> 9(vnet7) entered disabled state
> 2019-06-08T05:59:25.983702+02:00 ha-idg-1 systemd-machined[1383]: Machine 
> qemu-205-severin terminated.

The respective logs from libvirtd:
 ...
2019-05-31 20:31:34.481+0000: 4170: error : qemuMonitorIO:719 : internal error: 
End of file from qemu monitor
2019-06-01 01:05:32.233+0000: 4170: error : qemuMonitorIO:719 : internal error: 
End of file from qemu monitor
2019-06-01 01:05:43.804+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:05:43.848+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:06:11.438+0000: 26112: warning : 
qemuDomainObjBeginJobInternal:4865 : Cannot start job (destroy, none) for doma
in severin; current job is (modify, none) owned by (5372 
remoteDispatchDomainBlockJobAbort, 0 <null>) for (39s, 0s)
2019-06-01 01:06:11.438+0000: 26112: error : qemuDomainObjBeginJobInternal:4877 
: Timed out during operation: cannot acquire
state change lock (held by remoteDispatchDomainBlockJobAbort)
2019-06-01 01:06:13.976+0000: 5369: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:06:14.028+0000: 22596: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:06:44.165+0000: 5371: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:06:44.218+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:07:14.343+0000: 5369: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:07:14.387+0000: 22598: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-01 01:07:44.495+0000: 22605: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
 ...

2019-06-07 20:30:57.170+0000: 30299: error : qemuMonitorIO:719 : internal 
error: End of file from qemu monitor
2019-06-08 03:59:17.690+0000: 30299: error : qemuMonitorIO:719 : internal 
error: End of file from qemu monitor
2019-06-08 03:59:26.145+0000: 30300: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-08 03:59:26.191+0000: 30303: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-08 03:59:56.095+0000: 27956: warning : 
qemuDomainObjBeginJobInternal:4865 : Cannot start job (destroy, none) for 
domain severin; current job is (modify, none) owned by (13061 
remoteDispatchDomainBlockJobAbort, 0 <null>) for (38s,
 0s)
2019-06-08 03:59:56.095+0000: 27956: error : qemuDomainObjBeginJobInternal:4877 
: Timed out during operation: cannot acquire state change lock (held by 
remoteDispatchDomainBlockJobAbort)
2019-06-08 03:59:56.325+0000: 13060: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-08 03:59:56.372+0000: 30304: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data
2019-06-08 04:00:26.503+0000: 13060: warning : qemuGetProcessInfo:1461 : cannot 
parse process status data

The log from the domain itself (/var/log/libvirt/qemu/domain):

qemu-system-x86_64: block/mirror.c:864: mirror_run: Assertion 
`((&bs->tracked_requests)->lh_first == ((void *)0))' failed.

I'm running SLES 12 SP4 with qemu-2.11.2-5.8.1.x86_64 and kernel 4.12.14-95.13.


Bernd

-- 

Bernd Lentes 
Systemadministration 
Institut für Entwicklungsgenetik 
Gebäude 35.34 - Raum 208 
HelmholtzZentrum münchen 
address@hidden 
phone: +49 89 3187 1241 
phone: +49 89 3187 3827 
fax: +49 89 3187 2294 
http://www.helmholtz-muenchen.de/idg 

Perfekt ist wer keine Fehler macht 


Also sind Tote perfekt


Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling
Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, 
Kerstin Guenther
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671