[Monotone-devel] Binary files for benchmarking, and faster random file generation

[Eric Anderson]

I just pushed revision 9e7bf3f1355e0563551aad0fc22de3ef2b8033d7 which
adds in the ability to generate binary files in addition to text
files.  So that it is sane to generate large repositories, I also
added in a mkrandom.c program that can generate both the random binary
and text files.  My efforts to make it bit-compaitible with the python
code failed.  Despite the python and C programs both using the
mersenne twister RNG, they seem to generate a different pattern from
the same seed; I didn't invesitigate the python source code since it
wasn't clear this was an important feature.

The C program is 1.7-65x faster than the python version in the tests
shown at the bottom.  It's about 7-8x faster with the default tuning
constants for file sizes.  I suspect that if the random tree
generation code was modified to just write the files directly out to
their destination, rather than read them in to python and write them
back out that it could be faster to re-generate random trees than
cache them.
        -Eric

Timing for just the change() function in random_content.py:

num_files=100, filesize_logmean=12, filesize_logsd=0, binary_file_fraction=0.5:
   ELAPSED SECONDS TO BUILD:  1.38441491127 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  38.4065709114 -- without mkrandom

num_files=100, filesize_logmean=12, filesize_logsd=0, binary_file_fraction=0:
   ELAPSED SECONDS TO BUILD:  1.31179404259 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  85.58335495 -- without mkrandom

total_size=10000000, filesize_logmean=10, filesize_logsd=2, 
binary_file_fraction=0.5:
   ELAPSED SECONDS TO BUILD:  1.03478717804 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  30.5510389805 -- without mkrandom

total_size=5000000, filesize_logmean=8, filesize_logsd=2, 
binary_file_fraction=0:
   ELAPSED SECONDS TO BUILD:  2.02427911758 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  19.4277589321 -- without mkrandom

total_size=5000000, filesize_logmean=6, filesize_logsd=2, 
binary_file_fraction=0:
   ELAPSED SECONDS TO BUILD:  14.6136469841 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  25.1624319553 -- without mkrandom

total-size=5000000:
   ELAPSED SECONDS TO BUILD:  3.43155908585 -- with mkrandom
   ELAPSED SECONDS TO BUILD:  23.9149558544 -- without mkrandom

total-size=10000000:
   ELAPSED SECONDS TO BUILD:  6.96965098381
   ELAPSED SECONDS TO BUILD:  54.241093874