Re: [Monotone-devel] Improving client side performance of pulls

[Eric Anderson]

Nathaniel Smith writes:
 > On Wed, Aug 02, 2006 at 01:14:43AM -0700, Eric Anderson wrote:
 > > I was surprised there was so much time spent in lock and unlock, and
 > > by sampling some back traces in gdb, I found it seemed to be all
 > > memory allocation.
 > 
 > [ try using oprofile -c ]

If this was finding hot paths for the malloc/frees, it wasn't apparent
from the output.  It did make it clear that most of the memset() time
is coming from Botan.

 > > I see a number of possible solutions:
 > >   1) [ custom allocator, use vocab wrappers to make easier]
 > 
 > [ changes STL types, typedefs only help a little, but what about all
     the uses of std::string ]

Agreed.

 > >   2) [ non-unicode boost regex ]
 > 
 > [ unicode nice to have, may not be currently used ]

I'd expect that it's not currently used given that at least RHEL4
doesn't seem to ship with the ICU library that enables unicode support
for boost.

 > >   3) write a fake shim library that emulates all of the pthread calls, but
 > >      does nothing and aborts if something tries to create a thread. This
 > >      just seems wrong as an approach, but is possibly the simplest to 
 > >      implement.
 > 
 > Do we know if the locking is done by the STL allocators, or by
 > malloc/new themselves?  Because if it's the later, doesn't C++ provide
 > a hook to define a global replacement for operator new?  And
 > the same machinery that makes quick_alloc.h work must know how to
 > access some sort of low-level non-locking operator-new primitives?

The locking is happening in the malloc and free.

The following two links describe how to globally replace new and free,
but roughly seem to say don't do it.

http://www.awprofessional.com/articles/article.asp?p=30642&seqNum=3&rl=1
http://www.gotw.ca/gotw/010.htm

I still don't think any of these solutions are great.  I'm currently
implementing the shim library approach to see if eliminating the time
in the pthread routines will make it easier to track down where all of
the allocation is coming from.  Removing the allocation entirely would
be the best win if it's possible.
        -Eric