Re: Sort: buffer-size bug and nmerge performance issue

[Paul Eggert]

"William Herrin" <address@hidden> writes:

> 1. sort running on Linux kernel 2.6 x86_64 will accept but mishandle a
> --buffer-size argument larger than 2048M. It will silently pick some size
> noticeably smaller than 2048M. This is on a 64-bit machine with no ulimits
> on the amount of memory a process can consume.

First, you need to compile coreutils in 64-bit mode to be able to use
64-bit sizes.  (The default is to compile in 32-bit mode.)  I assume
you've done that?  If not, please try that.

Second, 'sort' is at the mercy of your malloc implementation.  That
is, 'sort' tries to malloc 2048M, and if that fails, it tries a
smaller allocation.  The idea is that it's better to get to sort done
than to complain about running out of memory.  Perhaps your C library
doesn't let 'malloc' succeed with large values?  You can test this by
writing a little test program.

> Has anyone done any testing to see if 16 is an optimal number for the merge
> size?

It depends on the platform.  It might well make sense to revisit that
number, or even make it a user option with a more-reasonable default.

Have you tried using compression on temporaries (the new
--compress-program option in coreutils 6.8)?  That probably affects
the optimal number.