Re: [Monotone-devel] Re: zlib vs gzip

[Markus Schiltknecht]

Hi,

Jack Lloyd wrote:
> Yes with that assumption I'd have to agree, but on anything else (eg,
> 4-way Opterons, old Unix machines with poor CPU and relatively
> powerful I/O, etc) it probably hurts quite a bit. Even on my Core2
> (with a RAID-1 setup, so each write has to hit all three disks), it is
> faster (wall clock) to write 200 Mb to disk than it is to first
> compress it with gzip and write the 30 Mb compressed result to disk.

Hm.. wow, yeah, on a very quick test, I got ~ 170 MB/s plain throughput, 
but only 13.5 MB/s when piping through gzip. Is compression really that 
slow even on modern hardware? I thought that changed long ago, well...

Thanks for pointing that out.

I remember the LZMA benchmarks [1], those seem to be more representative 
than my quick test. They list the following throughputs for an AMD 
mobile Athlon XP2400+ (in MB/sec):

                compression      decompression     compressed size (%)
gzip -1             18                61                  40,6
gzip -6              8.3              63                  37.2
gzip -9              3.0              65                  37.0
bzip2 -9             1.3               5.3                34.0
lzmash -9          < 0.27             20                  25.4

Thus, not even gzip -1 compression is close to a harddrive's throughput. 
Only gzip decompression comes somewhat close, probably making it quite a 
good fit.

I'm reminded of TOAST in PostgreSQL. They use a fast variant of an LZ 
compression algorithm. I'm wondering what throughput that achieves.

Regards

Markus

[1]: some LZMA benchmarks:
http://tukaani.org/lzma/benchmarks