Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

[Pádraig Brady]

On 28/02/10 00:46, Pádraig Brady wrote:
> On 27/02/10 22:16, Joey Degges wrote:
> > 2010/2/27 Pádraig Brady <address@hidden <mailto:address@hidden>>
> >
> > Well as long as the reads are big enough,
> > read-process-read-process-... won't be significantly different than
> > read-read-process-process-... However in the case where the
> > storage and CPU are contended for by other processes the former
> > is more desirable. You could test that by spinning the CPU with
> > another process while doing the test. It still seems to me
> > that WILLNEED is not appropriate for the sequential case.
> >
> > Here are the results of sorting when two other processes are running at
> > 100% each (dual core) for the duration of the sort:
> > NORMAL : 279.03s, 12.0% CPU
> > SEQUENTIAL: 230.92s, 14.5% CPU
> > WILLNEED : 186.26s, 18.5% CPU
> >
> > I suspect this trend may also repeat in situations where other processes
> > are contending for both CPU and storage. Perhaps the priority of the
> > WILLNEED read ahead is much higher than that of typical reads? Just a
> > guess.
>
> Interesting results, thanks!
> I'll play around with this with different buffer
> sizes and concurrent loads to see how it behaves
> on my single core system.

Will check WILLNEED v SEQUENTIAL this evening.

> > While we're looking at this it would also be worth considering
> > DONTNEED which could be used to drop the cache for temp files
> > when we're finished reading them. Note one shouldn't specify
> > a len of 0 until the file is completely read so as not to
> > invalidate any read ahead cache associated with the file.
> >
> >
> > Thanks for the suggestion. I have tested DONTNEED both in xfclose and in
> > fillbuf but do not see any improvements. It may be that this will only
> > improve performance in situations where resources are very tight. A
> > patch containing both WILLNEED and DONTNEED is included below.
>
> Yes DONTNEED wouldn't help sort unless resources are tight,
> but it might make sort impinge less on the system
> (though unlinking the temp files might have the same effect).

unlinking the temp files does nuke the cache as desired.
Also the unlinking happens in a timely manner as verified with:
  strace -e open,close,unlink sort -S1M <(seq 1000000) > /dev/null
So I'm dropping the DONTNEED part for the moment
(though there is the argument that temp files should be restricted
from using cache at all, as ideally they should only be used when
RAM is not available).

cheers,
Pádraig.