Re: Re: untar with plzip faster than with lzip for no apparent reason

[Sonia Diaz Pacheco]

wrotycz escribió:

I would appreciate if you do not include my email in a message for every spam not to read it.

Sure, count on it. Gmail adds the email address automatically, so I will delete it by hand.

I could not find it in source; it is hidden behind layers of abstractions but I did investigate it (in attachment) and you are actually right. Lzip does write 32 MB at a time, when plzip writes 1 MB at a time.

In the source file that decompresses from a regular file to stdout (dec_stdout.cc) the block size is defined in line 41:

   enum { max_packet_size = 1 << 20 };

What does --out-slots actually do I dont get at this point, as it seems to be in cotradiction to what I understand.

You may consider decreasing this to smaller value, like 1 MB, or so. On FreeBSD Wiki they tested it[1] and that turned out to give the best performance - 4 MB, and definitely 8 MB hurt performace equally to 64 kB.

That also, maybe coincidentally, happens to be max pipe size[2]. >From my test, it turned out that 7zip uses 1, 2, and with lzip 4 MB buffer, but have not seen more than that with any size of dictionary.

Other programs like xz use smaller buffer which hurts performance as well, this time from too small buffering, but that is another side of spectrum

--out-slots option determines the size of the store of blocks for each thread. As you can see in the manual:

--out-slots=n

Number of 1 MiB output packets buffered per worker thread when decompressing to non-seekable output. Increasing the number of packets may increase decompression speed, but requires more memory. Valid values range from 1 to 1024. The default value is 64.

Do not confuse the size of the block with the size of the store of blocks. Plzip by default leaves room to store up to 64 blocks per thread because, when decompressing a multimember file, all threads but one need to store their production until it can be sent to stdout.

Hope this helps.

Cheers,

Sonia