[Monotone-devel] RFC: Unit test improvements

[Zack Weinberg]

I've been back from my trip for a few days but I'm still pretty wrung
out and not up to doing much in the way of what you might call
_complicated_, so instead today I had a look at improving the way we
do unit tests.

There's three mostly-orthogonal improvements that I found:

* A facility for automatically registering test functions, inspired by
the handling of commands.  Instead of having to hand-write
"add_*_tests" functions you just do

UNIT_TEST(groupname, testname)
{
 ...
}

Group names substitute for the old grouping by registration function.
Groups are run in alphabetical order; within a group the order isn't
nailed down but seems (with GCC) to go in order of definition in the
file.

* Using boost::program_options (which was already being linked with
the unit_tests binary) I improved the argument handling.  You can now
ask for a list of supported tests or test groups, you can specify a
particular test within a group to be run, and you get sensible error
messages for command line arguments that don't correspond to tests or
options.  (Note: the test framework library claims to support a few
options but they don't work - I'm not sure whether this is because I
need to tell bpo to ignore them, or because the (nearly unreadable)
test framework documentation is just wrong.)

* And last but not least: Now that there are options, it is sensible
to turn debug-log-to-stderr off and require an option (--debug, just
like in monotone proper) to turn it back on.  Thus, the noise level of
make check is *dramatically* reduced.  I added fake tests to report
progress, it looks like this:

$ ./unit_tests
Running 96 test cases...
charset...
crypto...
cset...
diff_patch...
globish...
key...
netcmd...
packet...
paths...
pipe...
refiner...
restrictions...
mtn: warning: restriction includes unknown path 'foo'
mtn: warning: restriction excludes unknown path 'bar'
revision...
roster...
mtn: performing random action 0
mtn: performing random action 100
mtn: performing random action 200
mtn: performing random action 300
mtn: performing random action 400
mtn: performing random action 500
mtn: performing random action 600
mtn: performing random action 700
mtn: performing random action 800
mtn: performing random action 900
mtn: performing random action 1,000
mtn: performing random action 1,100
mtn: performing random action 1,200
mtn: performing random action 1,300
mtn: performing random action 1,400
mtn: performing random action 1,500
mtn: performing random action 1,600
mtn: performing random action 1,700
mtn: performing random action 1,800
mtn: performing random action 1,900
roster_merge...
simplestring_xform...
string_queue...
transform...
uri...
xdelta...

*** No errors detected

(I also hacked around the test-framework bug that makes the 'Running N
test cases...' message show up after everything else.  Arguably, we
should now eliminate the few messages that are still being printed,
due to use of P() instead of L(), but I can also argue that they
should be kept because the roster tests are so very slow.)

Thoughts?

zw

unit-test-improvements.diff
Description: Text Data