[Monotone-devel] nvm.experiment.db-compaction

[Markus Schiltknecht]

Hi,

during this easter hack, I've managed to get further with 
nvm.e.db-compaction. Rememder? The branch turning all database and 
in-memory ids into binary representation. It now passes all unit tests 
except schema_migration as well as those which are currently failing on 
mainline as well.

On my first attempt, which converted only the database ids, somebody 
objected that it would slow monotone down. Therfore I've moved on and 
converted all in-memory ids as well.

In the mean time, Zack committed a patch which allows monotone to 
recycle the botan pipes, which are used for hex en-/decoding among other 
things.

I've done some benchmarking, mainly with 
nvm.cbench/examples/annotate.sh. I've been using the following tests, 
with 20 repetitions, each.

    -b annotate-a='Annotate(ExistingRepo("../monotone.db"), 
"f97d974ae4670ce18c438613420d355fd8de80d4", "Makefile.am")' \
    -b annotate-b='Annotate(ExistingRepo("../monotone.db"), 
"f97d974ae4670ce18c438613420d355fd8de80d4", "paths.hh")' \
    -b annotate-c='Annotate(ExistingRepo("../monotone.db"), 
"f97d974ae4670ce18c438613420d355fd8de80d4", "xdelta.cc")' \
    -b annotate-d='Annotate(ExistingRepo("../monotone.db"), 
"f97d974ae4670ce18c438613420d355fd8de80d4", "database.cc")' \

I'm on an Intel EMT64 box, Core 2 Duo, SATA discs.

I've tested the following monotone revisions:

 * ref: a reference monotone binary from net.venge.monotone. Rev: 
a6e585755b93a0910cca494cd82d060333660ab5

 * dbc: from db-compaction branch, with all binary ids, but still uses 
L(FL(... encode_hexenc(..))) very often, which isn't conditional in this 
revision: 6e24d09551aabc0ca5bc7cda622b1d1eb4b4a76f

 * dbc_clog:  the conditional logging variant, where the L(FL( macro 
features an if (debug... before invoking the format parsing code. 
Especially here, that saves quite a bit of time.

Here are the results:

                          ann-a-ref ann-a-dbc ann-a-dbc_clog     p
annotate-avg-resident-MiB     23.36     23.24          23.20  0.63
    annotate-avg-size-MiB    119.30    119.68         119.80  0.06
annotate-max-resident-MiB     35.06     34.59          34.44  0.00
    annotate-max-size-MiB    132.07    131.56         131.52  0.00
     annotate-num-samples   2410.00   2398.05        2292.70  0.22
     annotate-system-time      0.86      1.19           1.17  0.00
       annotate-user-time      3.67      6.03           5.94  0.00
       annotate-wall-time     12.18     12.13          11.60  0.23
                          ann-b-ref ann-b-dbc ann-b-dbc_clog     p
annotate-avg-resident-MiB     12.69     12.48          12.47  0.01
    annotate-avg-size-MiB    106.35    106.99         106.88  0.02
annotate-max-resident-MiB     19.78     19.69          19.56  0.00
    annotate-max-size-MiB    116.86    116.72         116.75  0.00
     annotate-num-samples    762.50    757.50         696.00  0.52
     annotate-system-time      0.18      0.20           0.20  0.25
       annotate-user-time      0.51      0.79           0.77  0.00
       annotate-wall-time      3.87      3.84           3.53  0.52
                          ann-c-ref ann-c-dbc ann-c-dbc_clog     p
annotate-avg-resident-MiB     13.07     13.39          13.19  0.31
    annotate-avg-size-MiB    106.41    108.12         108.68  0.12
annotate-max-resident-MiB     19.46     19.54          19.39  0.00
    annotate-max-size-MiB    116.60    116.56         116.58  0.00
     annotate-num-samples    895.40   1005.50         837.00  0.72
     annotate-system-time      0.24      0.29           0.35  0.00
       annotate-user-time      0.74      1.21           1.18  0.00
       annotate-wall-time      4.56      5.09           4.25  0.72
                          ann-d-ref ann-d-dbc ann-d-dbc_clog     p
annotate-avg-resident-MiB     27.66     27.27          26.97  0.01
    annotate-avg-size-MiB    124.15    123.89         123.56  0.26
annotate-max-resident-MiB     36.85     36.44          36.36  0.00
    annotate-max-size-MiB    133.90    133.42         133.52  0.00
     annotate-num-samples   3531.80   3210.25        3199.20  0.00
     annotate-system-time      0.99      1.22           1.19  0.00
       annotate-user-time      7.84      9.56           9.42  0.00
       annotate-wall-time     17.86     16.24          16.18  0.00

To conclude: speed is somewhat better for annotate (didn't measure 
others, yet), while we also see a space saving of ... ehm.. what was it? 
8%? Don't remember exactly, at the moment. But the saving has been 
confirmed already. ;-)

Now, I'd like to land that branch on mainline as soon as feasible. I 
know the encode_hexenc() calls all over the place are annoying, but IMO 
we should revamp the vocab macros anyway. Holding of this binary id 
branch until then doesn't sound attractive to me.

I'm trying to fix the schema_migration test as soon as possible. That 
shouldn't prevent others from reviewing.

A small note: after db migrate, you need a db regenerate_caches, despite 
monotone not telling you... To be fixed with schema_migration...

Regards

Markus