bug#7868: splitting up test suites

[Stefano Lattarini]

Hello Ralf.

On Wednesday 19 January 2011, Ralf Wildenhues wrote:
> The testsuite is too large for MSYS.
>
Ouch.

> We've finally reached the point where we have more than 1000
> tests, $(TESTS) expands to 15k characters, and where 'make check' will
> not work at all any more on MSYS, because it cannot spawn sh any more,
> presumably in 'make check TESTS="..."'.  (MSYS make doesn't export
> macros to the environment of spawned processes even without .NOEXPORT,
> presumably otherwise lots of Makefiles would be really unusable here.)
> This also clears up the spurious failure of sed a few days ago.
>  
> Here's a preliminary plan for multiple testsuites per Makefile.am.
>
Hmmm... while this feature might be worth having even indipendently
from the issue at hand (but see below for small nits), I still think
that in the long run it would be nicer to transparently work around
such command-line length issues in the test driver, if possible.  Do
you think your patch "parallel-tests: avoid command-line length limit
issue" (from commit v1.11-191-g24e3b4e) could be resurrected in some
way?

> It would be nice if this worked somehow:
> 
> # These are all specified by the user:
> TEST_SUITE_LOGS = suite1.log suite2.log suite3.log ..
> TEST_EXTENSIONS = .test ...
> # these undergo $(EXEEXT) autoexpansion internally:
> suite1_log_SOURCES = all.test aclocal7.test ...
> suite2_log_SOURCES = suffix.test ...
>
What about using `TESTS' instead of `SOURCES' in these two last variables?
That would seems more natural to me, especially considering the API of the
current parallel-tests driver.

OTOH, would that maybe make the implementation more difficult?

> # These are then produced by automake:
> TEST_SUITE_HTMLS = $(TEST_SUITE_LOGS:.log=.html)
> suite1_log_LOGS = $(am__helper_var:.test=.log)
>
> mostlyclean-generic:
>       -test -z "$(suite1_log_LOGS)" || rm -f $(suite1_log_LOGS)
>       -test -z "$(suite2_log_LOGS)" || rm -f $(suite2_log_LOGS)
>
 
I think it would also be nice to generate separate check/recheck targets
for each testsuite; for example, "make check-suite1" could run all the
tests in $(suite1_log_SOURCES), and "make recheck-suite2" could re-run
all the tests in $(suite2_log_SOURCES) that failed (or xfailed) in the
previous run.

Hmm... No, wait, it would be even nicer to allow the user choose which
testsuite(s) to run by resetting the $(TEST_SUITE_LOGS) variable:

  make check TEST_SUITE_LOGS='suite1.log suite2.log'

Which makes me think that, perhaps, a variable like $(TEST_SUITES) would
be preferable:

  make check TEST_SUITES='suite1 suite2'

but then the API should be changed to something like:

 # These are all specified by the user:
 TEST_SUITES = suite1 suite2 suite3 ..
 TEST_EXTENSIONS = .test ...
 # these undergo $(EXEEXT) autoexpansion internally:
 suite1_TESTS = all.test aclocal7.test ...
 suite2_TESTS = suffix.test ...

 # These are then produced by automake:
 TEST_SUITE_LOGS = $(TEST_SUITES:=.log)
 TEST_SUITE_HTMLS = $(TEST_SUITES:=.html)
 suite1_LOGS = $(am__helper_var:.test=.log)

> In the <suite>_SOURCES, $(EXEEXT) transformation should take place
> (unless no-exeext is given, of course), just as is currently done for
> TESTS and *_PROGRAMS.  Hmm, alternatively we could also require all
> <suite>_SOURCES to be listed in $(TESTS), that would allow reuse of
> this variable as well, at the cost of some specification redundance.
>

> Open questions: how to produce nice results, both on stdout and in suite
> log files.  One way is to merge all logs into a final TEST_SUITE_LOG
> (that way we could also reuse that variable).  Another is to try to even
> hide the summaries of the individual suites, iff a final suite is being
> made.
>
These both sound sensible.  Another problem is how to avoid that, in a
parallel make run, the summary from a testsuite gets mixed and garbled
with the output/summary from another testsuite, as in e.g.:

 suite1_log_SOURCES = all.test aclocal7.test
 suite2_log_SOURCES = suffix.test yacc-dist-nobuild.test

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: aclocal7.test
 ====================================================
 PASS: suffix.test
 All 2 tests behaved as expected (1 expected failure)
 FAIL: yacc-dist-nobuild.test
 =====================================
 1 of 2 tests failed
 ====================================================
 See tests/suite2.log
 Please report to address@hidden
 =====================================

(yuck!), where we'd want at least:

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: suffix.test
 PASS: aclocal7.test
 ====================================================
 All 2 tests behaved as expected (1 expected failure)
 ====================================================
 FAIL: yacc-dist-nobuild.test
 =====================================
 1 of 2 tests failed
 See tests/suite2.log
 Please report to address@hidden
 =====================================

or better again:

 $ make check TEST_SUITE_LOGS='suite1.log suite2.log'
 XFAIL: all.test
 PASS: suffix.test
 PASS: aclocal7.test
 FAIL: yacc-dist-nobuild.test

 ==========================================================
 suite1: All tests behaved as expected (1 expected failure)
 ==========================================================

 =====================================
 suite2: 1 of 2 tests failed
 See tests/suite2.log
 Please report to address@hidden
 =====================================

> Internally, it would be nice if the register_language could be exploited
> (but this is not a requirement for this, as some features with tests are
> distinctly different from other languages).
> 
> Cheers,
> Ralf
> 

Thanks for tackling this!

Regards,
   Stefano