Re: feature request: gzip/bzip support for sort

[Dan Hipschman]

On Sat, Jan 13, 2007 at 10:36:05PM +0100, Jim Meyering wrote:
> Craig Macdonald <address@hidden> wrote:
> > On some occasions, I have the need to sort extremely large files, but
> > which compress well using programs such as gzip or bzip.
> ...
> 
> This task has been on the TODO list for some time:
> 
>   sort: Compress temporary files when doing large external sort/merges.
>     This improves performance when you can compress/uncompress faster than
>     you can read/write, which is common in these days of fast CPUs.
>     suggestion from Charles Randall on 2001-08-10
> 
> Just waiting for someone to work on it.

I already wrote a patch to sort quite a while ago that added compression
of the temporary files.  Unfortunately, with my code, I saw no
performance increase, so I discarded the patch.  However, if it can fix
the problem of running out of space, there may still be some use for it
after all.  The patch is old and needs some conflicts to be worked out,
but I'll get on it and submit ASAP.  In the meantime, I've posted the
original C code <http://linux.ucla.edu/~dsh/patches/compressed-sort.c>
in case you want to check it out and send comments.  It uses LZO
compression because it was supposed to be fast.  I also had patches to
the autoconf related stuff, but they seem to be hiding at the moment.
That part shouldn't be too hard to rewrite.