[Monotone-devel] net.venge.monotone.experiment.performance

[Eric Anderson]

I've created a net.venge.monotone.experiment.performance branch to put
a bunch of performance enhancing patches on them that may or may not
be appropriate for mainline.

I'll try to send a summary from time to time of the results, and I'm
trying to remember to put performance benchmarking in with each of the
updates.

Suitable for mainline:
  eddb7e59361efeb8d9300ba0ddd7483272565097:
    Make an upper bound on the amount of memory that will be consumed during
    a single commit.  Right now a commit will keep all of the compressed 
    differences in memory, which is not a good thing on a big import of
    an existing project.  Patch limits the amount stored in memory to 16MB,
    has no effect on sync because sync is flushing every 1MB.
    Detailed performance improvement included at the bottom since I forgot
    to include it in the commit message.

Needs further examination:
  997a677db676734acc0d098979d2a9cee8765ec9: 
    Add support for OpenSSL's libcrypto.  Measuring Pull(net.venge.monotone)
    on a 2.8GhZ Xeon, this was worth 1.3x cpu reduction on the server,
    and 2.4x cpu reduction on the client. On a 1.7GhZ Pentium-M, 1.07
    on the client, and almost nothing on the server.  If some other
    people could try this out (configure using --with-crypto to enable
    usage of the library), it would be nice to find out whether it matters
    on other platforms.  

  4e99cc37f548b5884d63c48bc486dfe98c8d0bd2:
    Add support for expidited parsing of rosters during annotation.  
    Also skip verification of SHA1 hashes only during annotation.
    Worth 5-20x speedup on annotation, but the faster parsing code may
    not succeed on all rosters that the standard code should parse.  I 
    belive the faster parsing code will abort in any case where it might
    do the wrong thing.

-Eric


Performance of the commit memory limiting patch; substantial on 
very large commits (728MB down to 325MB), moderate improvement
on medium sized things (105 down to 66), nothing on tiny things.

                     500MB-rand        500MB-txt        50MB-rand   
Statistic           perf     ref      perf    ref      perf    ref  
----------------- ------   ------   ------   ------  ------   ------
avg-resident-MiB   84.28 < 404.93    84.53 < 383.91   38.76 <  61.33
avg-size-MiB       99.50 < 420.10    99.44 < 398.70   53.62 <  76.23
max-resident-MiB  311.09 < 714.02   287.15 < 678.14   50.49 <  89.60
max-size-MiB      325.67 < 728.21   301.60 < 692.38   66.01 < 104.73
system-time         4.11 <   5.13     3.86 <   4.51    0.65 ?   0.64
user-time          75.98 ?  75.81    81.04 ?  81.15    9.77 ?   9.83
wall-time          89.00 <  91.36   102.37 ? 106.28   16.01 ?  15.72

                     50MB-txt        5MB-rand         5MB-txt
Statistic           perf     ref      perf    ref       perf   ref  
----------------- ------   ------   ------   ------   ------  ------
avg-resident-MiB   37.51 <  60.63    12.73 ?  11.98   11.99 ? 11.62
avg-size-MiB       52.49 <  75.70    26.89 ?  26.09   26.18 ? 25.77
max-resident-MiB   48.56 <  87.28    18.03 ?  18.11   17.04 ? 16.97
max-size-MiB       64.56 < 102.48    32.12 ?  32.14   31.07 ? 30.97
system-time         0.60 ?   0.57     0.07 ?   0.08    0.06 ?  0.07
user-time           9.95 ?  10.05     1.19 ?   1.18    1.19 ?  1.17
wall-time          15.09 ?  15.12     1.63 ?   1.32    1.42 ?  1.31