Re: [Monotone-devel] Patch to add Composite Benchmarks and some simple scalable add/commit tests

[Eric Anderson]

Nathaniel Smith <address@hidden> writes:
 > Subject: Re: [Monotone-devel] Patch to add Composite Benchmarks and
 >      some    simple scalable add/commit tests
 > 
 > On Mon, Jul 17, 2006 at 11:55:46PM -0700, Eric Anderson wrote:
 > > The idea behind the composite benchmark is that one can say:
 > > benchmark(setup-empty-workdir, setup-create-random-files, measure-add, 
 > >           measure-commit, measure-pull)
 > > 
 > > and the benchmark suite will first setup an empty working directory,
 > > create a bunch of random files in that working directory and then
 > > measure (using the instrumenter) the time to add, commit and pull the
 > > resulting repository.  This allows one to construct a series of
 > > operations and test them all in sequence.  The idea, although not
 > > implemented is that one could change measure- to setup- and the work
 > > would happen during setup not during measurement so that an expensive
 > > instrumenter would only be run over the relevant parts.
 > > 
 > > The precise CLI syntax for the above would look like:
 > >  -b simple-add='CompositeBenchmark(SetupEmptyWorkdir(), 
 > > SetupWorkdirAddFileSource(PartRandomFileSource(1,1024,0.5)), MeasureAdd(), 
 > > MeasureCommit(), MeasureWorkingPull())'
 > 
 > I still haven't had a chance to review the patch properly, sorry.
 > Having thought about it a bit, though, I'm feeling uncomfortable about
 > this general approach.  In particular, it is reminding me of the ad
 > hoc shell scripts that I was hoping to escape with this framework,
 > because it defines an oddly limited language for specifying procedural
 > computation (why am I writing a list of pre-defined steps, when I
 > could just be writing python code?), and it doesn't make explicit the
 > actual requirements (e.g., the Add benchmark needs a workspace, but
 > nothing here makes that obvious).  Finally, as a general note, I am
 > wary of tests that set up the environment for the next test, because
 > they are harder to reason about, harder to run individually or tweak
 > individually, and so on.

I'm confused about what you mean of an oddly limited language, you're
writing python, the stuff between the quotes is simply evaled by the
interpreter in the same way that you specify an instrumenter, etc.

I'm also a little confused about what you mean by wary of tests that
set up the environment for the next test.  How is that different from
having a whole series of setup steps and then a single measurement,
e.g. turning the above into:

 -b simple-add='AddToWorkspace(BuildWorkdir(Empty(),PartRandomFileSource(...)))'
 -b 
simple-commit='CommitToWorkspace(BuildWorkdir(Empty,PartRandomFileSource(...),AddFilesInWorkspace()))'

 -b simple-pull='Pull(BuildRepo(EmptyRepo,PartRandomFileSource(...)))'

These to me do the same thing with the big disadvantage of the bottom
ones that it is much harder to get the bottom pull benchmark to re-use
the work done in the commit and add.

 > The general idea of composing mix-and-match parts is very much where I
 > want to go, though!  The vague plan I've been working towards is
 > currently illustrated by the "Repo" interface, as implemented by
 > "EmptyRepo" and "ExistingRepo" and, hopefully soon,
 > "SimpleRandomRepo".  These all implement particular interface, which
 > is documented, and then the Pull benchmark requires a Repo.  So you
 > can say "Pull(EmptyRepo())" or "Pull(SimpleRandomRepo(treesize=10**5))"
 > or whatever.  I was planning to create a similar sort of interface for
 > benchmarks that require a workspace, or a workspace + a way to
 > generate edits in the workspace, etc.

I tried for a while to figure out how to work from what you had done
and failed.  I couldn't figure out how you were planning to handle
workspaces, and the existing code was incomplete.  Could you explain
how you want to handle a benchmark that starts with an empty
repository, adds in some zero filled, random filled, text filled
files; commits the changes, and then pulls the resulting repository?

 > The overall result is very similar to your example above, but I think
 > better factored -- basically the difference is instead of collapsing
 > things that set up environments and things that actually run tests
 > into one concept, trying to actually identify the different sorts of
 > environmental requirements different tests have, and implementing
 > those seperately.
 >
 > Of course, I haven't actually _done_ most of this yet, so who knows
 > if it will work.  But does that general Grand Plan make sense?

Not enough that I can translate it into what the code would look like,
I'm not even convinced I understand what the difference would be, I
put my best guess up above, and observed that it would be
significantly slower to run the benchmarks in that way than in the way
I did unless there's a lot of work on how to cache intermediates.

 > As for the specific case of porting existing tests, I think the
 > solution might be to just to create a way for tests to give names to
 > particular calls to the vcs object...?

What specific case is this referring to?
        -Eric