Re: [Bug-ed] GNU ed 1.10 released

[Antonio Diaz Diaz]

Hello Martin.

Martin Guy wrote:
> Certainly, for small systems, gzip is the compression of choice.

The lzip family is not a bad option for small systems. I test clzip and 
lunzip on a laptop with 4 MiB of RAM where modern gzip takes 5 and a 
half hours to 'configure && make && make check', and then just passes 10 
of the 15 tests. (It still has gzip 1.2.4 installed).

OTOH, lunzip just takes 3 minutes, clzip 9 minutes, and both pass all 
the tests. (Yes, 'clzip -9' works on 4 MiB of RAM for small files). (No, 
I do not test lzip on that laptop because its compiler is very old; gcc 
2.7.2).

BTW. Given that gzip's configure takes more than one hour to run on that 
laptop while clzip's takes just a few seconds, comments like "Upstream 
configure script is moronic"[1] show how clueless distros are.

[1]http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/sys-apps/ed/ed-1.10.ebuild?view=markup


> To unzip gmp-snapshot, gzip takes 600Kbyte of RAM where lunzip takes
> 15MB, and lunzip takes twice as much CPU time to decompress. The same
> was true when bzip2 was the preferred power compression.

Lunzip takes 15 MB to decompress gmp-snapshot because this is the 
dictionary size used to compress the file. Just compress the file using 
'lzip -0' and lunzip will take only 80 KiB of RAM to decompress it.

Of course, using a smaller dictionary size reduces the compression 
ratio. There is no such thing as a free lunch:
  3054541 gmp-5.1.90-20140224.tar0.lz
  3081815 gmp-5.1.90-20140224.tar.gz


> For example, the fact that one package is now lzipped instead of
> gzipped means that you now cannot install Debian on a system with 64MB
> of RAM because one package fails for lack of VM during decompression.

According to 'valgrind --tool=massif' Lzip needs about 528 KiB to 
decompress ed-1.10.tar.lz. This is much less than bzip2 (3.5 MiB), and 
two orders of magnitude less than xz (64 MiB). All of them compressed at 
level -9.

Sadly massif can't measure the amount of memory used by gzip because it 
does not seem to use the heap. (Massif says it uses just 7 KiB).


> However, I see that the compression achieved by lzip is almost twice
> as good as gzip:
>  3081791 gmp-5.1.90-20140224.tar.gz
>  1898483 gmp-5.1.90-20140224.tar.lz

And 1% better than xz, while using four times less memory (at level -9).
  1916488 gmp-5.1.90-20140224.tar.xz


> so I guess that the reasons may be purely practical: to save on GNU's
> astronomical bandwidth costs for downloads. That is compelling for
> large packages or packages that can only be used on powerful systems
> anyway, less so for ed, which is small in itself and more likely to be
> used on small systems that are more likely to be able to run gzip.

As shown above, lzip can be used on small systems. If one format is to 
be used for large and small files on large and small systems, lzip is 
the only one that fits all uses.


Best regards,
Antonio.