Re: HOWTO: benchmarking monotone (was Re: [Monotone-devel] "memory exhausted" error for 'mtn list status' command)

[Eric Anderson]

Nathaniel Smith writes:
 > On Fri, Jul 28, 2006 at 11:58:29AM -0700, Eric Anderson wrote:
 > > avg resident size has decreased from 4.35 to 3.1
 > > MiB, avg size has decreased from 21.4 MiB to 19.8 MiB. Similar on the
 > > maximum values.  The only odd thing is that all three "after" runs
 > > took way more wall clock time, but similar system and user time.
 > 
 > Hrm, weird -- I actually re-ran these when writing the email, since
 > I'd deleted my first run, and I could swear the numbers were weirder
 > before :-).  (Weirder, like, swap the average resident sizes between
 > the two runs.)  I still have some trouble interpreting these -- why are
 > the max-resident sizes the same?  

I would imagine because there is some point in time where both
processes have the same resident size; it could be at the beginning
Botan does a remarkable amount of self-testing on every run.

 > I would have thought that was the
 > most interesting number, since peak memory usage is often more
 > important than average memory usage, and the total (as opposed to
 > resident) numbers are clearly counting, like, the size of all linked
 > libraries or something.

 > Probably I/we just need more experience in interpreting these numbers;
 > they're a little trickier to understand than I expected.  (It might
 > still be interesting to add an instrumenter that hooks into malloc and
 > gives peak _heap_ amounts in particular, just because those are
 > presumably less noisy and easier to interpret in terms of code.)

The memtime program is doing sampling of the values in /proc to figure
out how much memory is used at any particular time.  This means that
it can miss very short peaks.  In general I ignore differences of <
10%, so would have considered the two runs you had to be pretty much
the same, even though there is a valid statistical difference on both
the avg-{resident,size} and max-size.  That said a resident size of
<10MB is pretty much nothing.
        -Eric