Re: [PATCH] sort: Add --threads option, which parallelizes internal sort.

[Ralf Wildenhues]

Hi Glen,

* glen lenker wrote on Fri, Jun 19, 2009 at 01:03:10PM CEST:
> On Fri, Jun 12, 2009 at 12:54 AM, Ralf Wildenhues wrote:
> >
> > No, I did not specify the amount of RAM.  The system I tested on has
> > plenty of RAM, way more than is needed for the sort.  Specifying
> > something like 2G of RAM does not make any visible difference.
> 
> When testing on gcc16, 16G of RAM, sort chooses to use only 1G of
> RAM for a 256M file, and ends up hitting the external merge code. It
> was through experimentation with gprof that I found that anything less
> than 2G causes sort to do an external sort on gcc16.

Hmm, ok.  I did retest with 2G at the time, but I didn't check logs for
whether the external merge code was used.  I can try to recheck sometime
(systems are busy ATM).

> > > > 160.22user 1.34system 2:41.61elapsed 99%CPU
> > > > 159.83user 1.45system 1:27.12elapsed 185%CPU
> > > > 159.84user 1.56system 0:52.26elapsed 308%CPU
> > > > 160.67user 1.53system 0:36.26elapsed 447%CPU
> > > >
> > > > This seems to be what I would expect from a good implementation.
> >
> > Yes, 56% efficiency going from 1 to 8 threads sounds like a much better
> > number, also your system overhead looks very sane compared to what I
> > saw.  Seems like it's a system-specific issue after all.  Which Linux
> > kernel version and which pthread (glibc) version are you using?
> 
> This was on gcc16.
> Linux gcc16 2.6.18-6-vserver-amd64 #1 SMP Thu Dec 25 21:35:11 UTC 2008
> x86_64 GNU/Linux
> glibc 2.3.6.ds1-13etch9

here:
Linux 2.6.24-23-generic
GNU C Library stable release version 2.7, by Roland McGrath et al.

> > Glen, could you look at this, too?  I mean just timings of 1 vs 2
> > threads for different file sizes?  Thanks.
> 
> The data was again cat'ed instances of the dictionary, put onto tmpfs. I
> made sure that each time it had enough memory.
> 
> 1st run
> memory    1 thread    2 threads   speedup
> 16M          7.27         4.04           ~1.8
> 32M          16.15        9.19          ~1.75
> 64M          35.06        19.61         ~1.78
> 128M        75.47        40.99         ~1.84
> 256M        162.30       87.27         ~1.86
> 
> 2nd run
> memory   1 thread    2 threads    speedup
> 16M        7.27          4.05            ~1.8
> 32M        16.15         8.86           ~1.82
> 64M        35.07         19.18          ~1.83
> 128M      75.22         41.28          ~1.82
> 256M      161.76       87.77           ~1.84

Well, that is a small increase and probably below noise level,
but at least there is no decrease, and it looks fairly consistent,
so I think that is good.

> On this subject, what exactly is the motivation of sort's
> default_sort_size()?
> Shouldn't it try to do an internal sort if possible?

Reading the change logs, I think it was added to avoid overcommitting
memory (and thus hitting the chance of being killed later by an OOM
situation).

Cheers,
Ralf