Sort: buffer-size bug and nmerge performance issue

[William Herrin]

Hi folks:

One bug and one performance issue for the sort command in Coreutils 6.7.
This also applies to Coreutils 5.94. Results were observed with a freshly
compiled copy of sort on an x86_64 machine running Red Hat Enterprise Linux
AS release 4, kernel 2.6.9.

1. sort running on Linux kernel 2.6 x86_64 will accept but mishandle a
--buffer-size argument larger than 2048M. It will silently pick some size
noticeably smaller than 2048M. This is on a 64-bit machine with no ulimits
on the amount of memory a process can consume.

2. In sort, NMERGE is set to 16. When sort runs out space in its buffer, it
stores sorted pieces in /tmp and then performs a merge on the pieces. If
there are more than 16, it has to repeat this process, merging sets of 16
into larger files, then mergeing those larger files again, etc.

Has anyone done any testing to see if 16 is an optimal number for the merge
size? My own ad-hoc tests suggest than a setting closer to 128 or 256 allows
sort to finish much sooner and consume fewer resources on the machine.

Regards,
Bill Herrin


--
William D. Herrin                  address@hidden   address@hidden
3005 Crane Dr.                        Web: < http://bill.herrin.us/>
Falls Church, VA 22042-3004