Re: Compressed man pages (was: Accessibility of man pages (was: Playground pager lsp(1)))

[Alejandro Colomar]

Hi all,

After the suggestion by Ralph of trying .lz, Sam's comment about .xz),
and Kerin's comment about tuning the compression parameters, I decided
to try out everything at once, so we can see the effects of the
alternatives.

TL;DR:  For manual pages, use uncompressed source, or gzip(1).
        Everything else is unreasonably slow.


Here go the numbers.  Below, will be a conclusion I get from them.
The following tests have been produced with man-db's man(1) built
from source, since Colin fixed an relevant bug a few days ago[1].
This improves performance considerably compared to the latest
release.


$ sudo make install-man prefix=/opt/local/man/bz2_1 -j LINK_PAGES=symlink 
Z=.bz2 BZIP2FLAGS=-1 | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/bz2_9 -j LINK_PAGES=symlink 
Z=.bz2 BZIP2FLAGS=-9 | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/bz2__ -j LINK_PAGES=symlink 
Z=.bz2 BZIP2FLAGS=   | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz__1 -j LINK_PAGES=symlink Z=.gz 
 GZIPFLAGS=-1  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz__9 -j LINK_PAGES=symlink Z=.gz 
 GZIPFLAGS=-9  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/gz___ -j LINK_PAGES=symlink Z=.gz 
 GZIPFLAGS=    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz__1 -j LINK_PAGES=symlink Z=.lz 
 LZIPFLAGS=-1  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz__9 -j LINK_PAGES=symlink Z=.lz 
 LZIPFLAGS=-9  | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/lz___ -j LINK_PAGES=symlink Z=.lz 
 LZIPFLAGS=    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz__1 -j LINK_PAGES=symlink Z=.xz 
 XZFLAGS=-1    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz__9 -j LINK_PAGES=symlink Z=.xz 
 XZFLAGS=-9    | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/xz___ -j LINK_PAGES=symlink Z=.xz 
 XZFLAGS=      | wc -l
2571
$ sudo make install-man prefix=/opt/local/man/man__ -j LINK_PAGES=symlink Z=    
               | wc -l
2571
$ du -sh /opt/local/man/*
5.4M    /opt/local/man/bz2_1
5.4M    /opt/local/man/bz2_9
5.4M    /opt/local/man/bz2__
5.7M    /opt/local/man/gz__1
5.5M    /opt/local/man/gz__9
5.5M    /opt/local/man/gz___
5.5M    /opt/local/man/lz__1
5.4M    /opt/local/man/lz__9
5.4M    /opt/local/man/lz___
9.4M    /opt/local/man/man__
5.5M    /opt/local/man/xz__1
5.4M    /opt/local/man/xz__9
5.4M    /opt/local/man/xz___


$ export MANPATH=/opt/local/man/bz2_1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.15
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.22

$ export MANPATH=/opt/local/man/bz2_9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.15
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.23

$ export MANPATH=/opt/local/man/bz2__/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.19
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do bzip2 -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.23


$ export MANPATH=/opt/local/man/gz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.21
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.16

$ export MANPATH=/opt/local/man/gz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.20
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.17

$ export MANPATH=/opt/local/man/gz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.20
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do gzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.15


$ export MANPATH=/opt/local/man/lz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.95
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.40

$ export MANPATH=/opt/local/man/lz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.93
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.40

$ export MANPATH=/opt/local/man/lz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.94
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do lzip -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.40


$ export MANPATH=/opt/local/man/xz__1/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 3.43
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.24

$ export MANPATH=/opt/local/man/xz__9/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 4.21
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.55

$ export MANPATH=/opt/local/man/xz___/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 4.17
$ /bin/time -f %e dash -c "find $MANPATH -type f | while read f; do xz -d - 
<\$f | grep -l RLIMIT_NOFILE >/dev/null && echo \$f; done | wc -l | xargs 
printf '%s; '"
17; 1.55


$ export MANPATH=/opt/local/man/man__/share/man
$ /bin/time -f %e dash -c "man -Kaw RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.55
$ /bin/time -f %e dash -c "find $MANPATH -type f | xargs -P0 grep -l 
RLIMIT_NOFILE | wc -l | xargs printf '%s; '"
17; 0.01


Conclussions:

Any compression formats other than .gz are unreasonably slow.
I'd say either use .gz, or plain text, or prepare to
contribute code yourself to man-db to optimize for your favourite
compression format.

.bz2, .lz, and .xz have similar times, and tuning the compression
doesn't produce important changes in speed (except slightly for
.xz, but I don't see any advantage of using .xz).

Similarly, tuning the compression of .gz doesn't produce
important changes in speed.

Plain text has the advantage that you can use all the power of
Unix tools to search through the source code of the pages
instantaneously, without being restricted to what man(1) allows.


I hope this was useful.

Cheers,
Alex


[1]:  <https://lists.nongnu.org/archive/html/man-db-devel/2023-04/msg00000.html>

-- 
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5

OpenPGP_signature
Description: OpenPGP digital signature