Re: [Monotone-devel] net.venge.monotone.experiment.performance

[Eric Anderson]

Nathaniel Smith writes:
 > [ benefit of writing files/rosters is not writing full-text versions
     that will be deleted; perhaps defer compression ]

Oh, that makes much more sense, I'd imagine you could defer the
compression, but according to my oprofile measurements the compression
piece isn't yet a dominant factor.

 > removed the comments about the write buffer
 > size vs. the netsync commit interval, which are logically separate.
 > (The netsync commit interval is mostly tuned to choose how much we
 > spend waiting for the disk to fsync(), while the write buffer size is
 > tuned to choose how much stuff we end up with in memory; also, they're
 > not even measuring the same bytes, really -- netsync looks at bytes
 > transferred, while the write cache includes reconstructed full
 > texts...)

You're right, I thought that the checkpoint batch size was using the
size of things written to the database, but it looks like it's using
the amount of stuff that came over the wire.

 > In the future, it would be convenient if you could commit different
 > changes in parallel rather than sequence (possibly even as different
 > branches), so that they could be merged individually; as it was, I had
 > to use pluck and remove some extraneous changes, so you may have some
 > extra work merging back to .performance.

So having a multi-headed branch is an interesting suggestion.  I
thought about it initially and decided against for two reasons:
  1) I'm trying to build an optimized version of monotone that I can
     release internally that will go faster than the last version I 
     optimized.
  2) When optimizing I find that once you optimize a few things, you
     can get the time down enough that you can find something else 
     that's taking up a lot of the time, but was hidden by noise 
     caused by other parts of the system.  

As an example of (2), I've finally tracked down that a huge amount of
time (~30%) is being spent during a pull writing out the rosters.  I
was able to tell because I'd optimized enough that a run of callgrind
was plausible (it still took 15 hours), and when I looked at the
results I wasn't spending time looking at the other overhead that was
easily identifyable with oprofile.

Conversely, your goal is to pick out the subset of fixes that I make
that are good enough for mainline. I'll hope you'll also use the
combination of all of them as a comparison point in benchmarking.

One possibility I thought of to achieve both our goals would be to
have a nvmep.merged and nvmep.individual; avoiding the problem of
having 10-50 branches, but it turns out that propogate won't work with
multiple heads.  I generally try to avoid lots of branches because it
makes the output of mtn list branches and viewmtn much harder to read,
especially since most of these branches would have 1 or maybe 2
commits on them and then would be done, but I could go down that path
if that's the best solution.

I wonder though if this isn't an interesting use-case for monotone
that would be worth supporting more directly, either by allowing a
propogate from an unmerged branch, or by allowing the definition of a
branch that is made up of merging a number of other revisions/branches
in some order (similar to quilt as I understand it from skimming the
documentation).  Thoughts?
        -Eric