Re: Big TCG slowdown when using zstd with aarch64

[Juan Quintela]

Daniel P. Berrangé <berrange@redhat.com> wrote:
> On Thu, Jun 01, 2023 at 11:06:42PM +0200, Juan Quintela wrote:
>> 
>> Hi
>> 
>> Before I continue investigating this further, do you have any clue what
>> is going on here.  I am running qemu-system-aarch64 on x86_64.
>> 
>> $ time ./tests/qtest/migration-test -p 
>> /aarch64/migration/multifd/tcp/plain/none
>
>
>> real 0m4.559s
>> user 0m4.898s
>> sys  0m1.156s
>
>> $ time ./tests/qtest/migration-test -p 
>> /aarch64/migration/multifd/tcp/plain/zlib
>
>> real 0m1.645s
>> user 0m3.484s
>> sys  0m0.512s
>> $ time ./tests/qtest/migration-test -p 
>> /aarch64/migration/multifd/tcp/plain/zstd
>
>> real 0m48.022s
>> user 8m17.306s
>> sys  0m35.217s
>> 
>> 
>> This test is very amenable to compression, basically we only modify one
>> byte for each page, and basically all the pages are the same.
>> 
>> no compression: 4.5 seconds
>> zlib compression: 1.6 seconds (inside what I would expect)
>> zstd compression: 48 seconds, what is going on here?
>
> This is non-deterministic. I've seen *all* three cases complete in approx
> 1 second each. If I set 'QTEST_LOG=1', then very often the zstd test will
> complete in < 1 second.

Not in my case.
/me goes and checks again.

Low and behold, today it don't fails.

Notice that I am running qemu-system-aarch64 in x86_64 host.
Yesterday I was unable to reproduce it with kvm x86_64 in x86_64 host.

And for aarch64 I reproduced it like 20 times in a row, that is why I
decided to send this email.

In all the other cases, it behaves as expected.  This is one of the few
cases where compression is way better than not compression.  But
remember that compression uses dictionaries, and for each page it sends
the equivalent of:

1st page: create dictionary, something that represents 1 byte with value
          X and TARGET_PAGE_SIZE -1 zeros
Next 63 pages in the packet: copy the previous dictionary 63 times.

I investigated it when I created multifd-zlib because the size of the
packet that described 64 pages content was ridiculous, something like
4-8 bytes (yes, I don't remember, but it was way, way less that 1 byte
per page).

> I notice the multifd tests are not sharing the setup logic with the
> precopy tests, so they have no set any migration bandwidth limit.
> IOW migration is running at full speed.

Aha.

> What I happening is that the migrate is runing so fast that the guest
> workload hasn't had the chance to dirty any memory, so 'none' and 'zlib'
> tests only copy about 15-30 MB of data, the rest is still all zeroes.
>
> When it is fast, the zstd test also has similar low transfer of data,
> but when it is slow then it transfers a massive amount more, and goes
> through a *huge* number of iterations
>
> eg I see dirty-sync-count over 1000:

Aha, will try to print that info.

> {"return": {"expected-downtime": 221243, "status": "active",
> "setup-time": 1, "total-time": 44028, "ram": {"total": 291905536,
> "postcopy-requests": 0, "dirty-sync-count": 1516, "multifd-bytes":
> 24241675, "pages-per-second": 804571, "downtime-bytes": 0,
> "page-size": 4096, "remaining": 82313216, "postcopy-bytes": 0, "mbps":
> 3.7536507936507939, "transferred": 25377710,
> "dirty-sync-missed-zero-copy": 0, "precopy-bytes": 1136035,
> "duplicate": 124866, "dirty-pages-rate": 850637, "skipped": 0,
> "normal-bytes": 156904067072, "normal": 38306657}}}
>
>
> I suspect that the zstd logic takes a little bit longer in setup,
> which allows often allows the guest dirty workload to get ahead of
> it, resulting in a huge amount of data to transfer. Every now and
> then the compression code gets ahead of the workload and thus most
> data is zeros and skipped.

That makes sense.  I think that the other problem that I am having this
days is that I am loading my machine a lot (basically running
make check in both branches at the same time, and that makes this much
more probable to happens.)

> IMHO this feels like just another example of compression being largely
> useless. The CPU overhead of compression can't keep up with the guest
> dirty workload, making the supposedly network bandwidth saving irrelevant.

I will not say that it make it useless.  But I have said since quite a
long time that compression and xbzrle only make sense if you are
migration between datacenters.  Anything that is in the same switch, or
that only needs a couple of hops in the same datacenter it makes no
sense.

Later, Juan.