Re: [ovirt-users] Possible sources of cpu steal and countermeasures

[Klaas Demter]

Hi,

Thanks for the quick response.

On Thu, Dec 5, 2019, 01:10 Nir Soffer <address@hidden> wrote:

On Wed, Dec 4, 2019 at 6:15 PM <address@hidden> wrote:
>
> Hi,
>
> I'm having performance issues with a ovirt installation. It is showing
> high steal (5-10%) for a cpu intensive VM. The hypervisor however has
> more than 65% of his resources idle while the steal is seen inside of
> the VM.
>
> Even when placing only a single VM on a hypervisor it still receives
> steal (0-2%), even though the hypervisor is not overcommited.
>
>
> Hypervisor:
>
> 2 Socket system in total 2*28(56HT) cores
>
>
> VM:
>
> 30vCPUs (ovirt seems to think its a good idea to make that 15 sockets *
> 2 cores)

I think you can control this in oVirt.

Yes, I'm saying ovirt defaults to using 15 sockets * 2 cores if you don't specify a different "advanced" configuration. This also leads to a funny behavior that you can't increase the VMs from 30 to 35 cores online.

> My questions are:
>
> a) Could it be that the hypervisor is trying to schedule all 30 cores on
> a single numa node, ie using the HT cores instead of "real" ones and
> this shows up as steal?
>
> b) Do I need to make VMs this big numa-aware and spread the vm over both
> numa nodes?
>
> c) Would using the High Performance VM type help in this kind of situation?
>
> d) General advise: how do I reduce steal in an environment where the
> hypervisor has idle resources
>
>
> Any advise would be appreciated.

These questions are mainly about qemu, so adding qemu-discuss.

I think it will help if you share your vm qemu command line, found in:
/var/log/libvit/qemu/vm-name.log

Log:

2019-11-16 22:33:33.487+0000: starting up libvirt version: 4.5.0, package: 10.el7_6.6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-02-21-14:30:01, x86-038.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-18.el7_6.3, kernel: 3.10.0-957.10.1.el7.x86_64, hostname: hypervisor.tld

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=vm,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-24-vm/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu Broadwell,spec-ctrl=on,ssbd=on,rtm=on,hle=on -m size=534773760k,slots=16,maxmem=629145600k -realtime mlock=off -smp 30,maxcpus=32,sockets=16,cores=2,threads=1 -object iothread,id=iothread1 -numa node,nodeid=0,cpus=0-29,mem=522240 -uuid 95285008-8efb-446c-b75c-771f020cf494 -smbios 'type=1,manufacturer=oVirt,product=RHEV Hypervisor,version=7.6-4.el7,serial=4c4c4544-004b-3510-805a-c3c04f485032,uuid=95285008-8efb-446c-b75c-771f020cf494' -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=51,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2019-11-16T22:33:33,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,iothread=iothread1,id=ua-98c9be1a-c830-4718-9936-28a81e9a07b6,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,id=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,werror=report,rerror=report,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b,id=ua-641b4623-d2b3-4f7f-a31d-65959f5ffc9b -drive file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads -device scsi-hd,bus=ua-98c9be1a-c830-4718-9936-28a81e9a07b6.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,id=ua-88901d29-e223-457d-a80e-384b1f147cea,bootindex=1,write-cache=on -netdev tap,fds=54:55:56:57,id=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,vhost=on,vhostfds=58:59:60:61 -device virtio-net-pci,mq=on,vectors=10,host_mtu=1500,netdev=hostua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,id=ua-1fad81fe-8c0b-41dd-8015-7498ebbc2710,mac=22:44:66:00:00:d0,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,fd=62,server,nowait -device virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 -chardev socket,id=charchannel1,fd=63,server,nowait -device virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=ua-3f6418a3-8b1a-49c3-b79b-ca5f1864cf47.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5918,tls-port=5919,addr=10.160.69.12,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=ua-a3bde96a-2165-434e-80f5-5980dbb9f19d,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=ua-124c48f3-6bc4-453c-b0fb-107060bbc1f6,bus=pci.0,addr=0x6 -object rng-random,id=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,filename=/dev/urandom -device virtio-rng-pci,rng=objua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,id=ua-ff7eb44a-85a1-4731-aec0-0b85b16c2644,bus=pci.0,addr=0x7 -device vmcoreinfo -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

2019-11-16T22:33:33.624932Z qemu-kvm: -drive file=/rhev/data-center/mnt/storage-domain/0f623c51-66c8-4a5b-a12f-95ab9c6184b4/images/88901d29-e223-457d-a80e-384b1f147cea/0f994c76-d525-4a4f-9513-c6220fe7c31c,format=raw,if=none,id=drive-ua-88901d29-e223-457d-a80e-384b1f147cea,serial=88901d29-e223-457d-a80e-384b1f147cea,werror=stop,rerror=stop,cache=none,aio=threads: 'serial' is deprecated, please use the corresponding option of '-device' instead

2019-11-16T22:33:33.630420Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 30 [socket-id: 15, core-id: 0, thread-id: 0], CPU 31 [socket-id: 15, core-id: 1, thread-id: 0]

2019-11-16T22:33:33.630439Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future

Nir