Re: [PATCH] sort: Add --threads option, which parallelizes internal sort.

[Jim Meyering]

Ralf Wildenhues wrote:

> * Jim Meyering wrote on Thu, May 28, 2009 at 09:33:21PM CEST:
...
>> I ran some tests (see below), and got results that look
>> similar to Ralf's.  I used an otherwise-idle 16-core Nehalem-based
>> system with 8GB of RAM.
>
> Which kernel and glibc?

admittedly old kernel: 2.6.18-128.el5xen
and glibc-2.5-34

>> However, more importantly, while the 16- and 8-thread tests were
>> running, I sampled thread counts using ps and was surprised to see
>> the number of active threads was usually far less than the maximum.
>
> Well yes, I suppose you are seeing the serial overhead from reading in
> the data set by the first process alone, and from the later merging
> stages where only fewer threads are active.
>
>>   T=16             T is number of threads
>>   12.96       <--- This is elapsed wall-clock time in seconds.
>>   16.43
>>   13.76
>>   16.60
>
> That varies a lot.  Are there other jobs on this system running?
> Do you, Jim, also see high system time overhead?  That would support
> the hypothesis of a less efficient thread implementation (or kernel
> issue) being the cause.

Here are more numbers, this time including user and system times, too:

for t in 16 8 4 2 1; do echo T=$t; for i in 1 2 3 4; do env time --format '%e 
%U %S' sort --threads=$t --buffer-size=2G < in > /dev/null; done; done
T=16
16.56 54.70 1.53
12.88 50.60 1.38
16.32 55.12 1.48
13.77 50.72 1.56
T=8
14.49 42.72 1.54
17.47 46.15 1.47
14.99 44.59 1.52
15.05 43.50 1.49
T=4
17.32 38.37 1.29
20.27 44.46 1.46
20.82 45.15 1.44
20.01 41.01 1.54
T=2
26.74 39.02 1.48
27.97 43.93 1.44
26.69 39.13 1.34
27.94 43.80 1.51
T=1
43.28 41.77 1.50
43.24 41.67 1.56
43.17 41.69 1.48
43.23 41.87 1.35