Following the bisect thread brought some questions about testresult back to mind. This was one of the features of monotone that drew me to it, back around 0.14, and I've never really gotten around to really implementing testresult in my code projects.
Is there a best practice on using the mtn testresult command? I've been poking at the idea of a CI-style monotone bot that would update a project, run the associated test suite, and mark the current revision with testresults. I keep butting up against problems of suite changes:
* If I change how a test works, are it's results valid on past revisions, or do I need to re-run that test against past revisions? * If I change the name of a test, is that trackable? Won't that break acceptance of future revisions, since there's an old "pass" value that isn't on current revisions? (Changeable in Lua, obviously)
* Is it reasonable to use monotone style testresults and keep tests and code together?