bug#11287: Various issues with the test suite framework

[Akim Demaille]

Le 24 avr. 2012 à 11:42, Stefano Lattarini a écrit :

> On 04/20/2012 01:13 PM, Akim Demaille wrote:
>> Hi!
>> 
> Hi Akim, sorry for the delay.

Hi Stefano,

No worries.

>> I have seen that check-html will be removed, or rather moved
>> into the contrib part, but there are a few issues:
>> 
>> - the target is not declared recursive, so one has to write
>> the bouncing target herself.
>> 
>> - because of that, the "naive" implementation of check-html
>> that just bounces the right directories will miss the
>> dependencies that check features (in other words, make
>> check properly recursively builds before, while check-html
>> would just go into the typical tests/ directory, and then
>> bounce to "check", so it completely missed the recursion
>> in the other directories and their dependencies).
>> 
> Wasn't this issue already present in Automake 1.11.1?  (So, not a
> regression, unless I'm mistaken).  In any case, a test case
> exposing the issue would be appreciated, if you have one at hand.

No, I don't think this is a regression.  It is a problem though.
I don't know (yet) what contribs/ are in Automake, but this shows
that some real assistance from Automake would really help, it's
not just a matter of a single Make snippet in a single Makefile,
but something more pervasive to the whole package.

>> - the .log.html recipe features $$RST2HTML, but in '', so it is
>> useless.  And anyway $(RST2HTML) already does the job.
>> 
>> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>> 
> None of these are covered by the Automake test suite sadly, and I
> see them as really low low priority anyway, so I'm not going to
> tackle them right now.  But I'll gladly accept patches by anyone
> who has more knowledge or interest than me in this area ;-)

I'll look for someone matching these specs :)


>> - RECURSIVE_TARGETS cannot be highjacked by the users, that's
>> sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
>> RECURSIVE_TARGETS so that we can add our own targets that
>> traverse the whole package?
>> 
> There had been an attempt to offer an interface for user-defined
> recursive rules already:
> 
>  <http://thread.gmane.org/gmane.comp.sysutils.automake.general/11890>
> 
> But unfortunately the discussion stagnated and died out, and I don't
> have time to resurrect it nor the relevant patches ATM.  Still, it
> would be a cool feature to have, so any help would be appreciated.

Yes, definitely.

>> - the test compiler does not appear as a dependency for
>>  the test logs that use it.
>> 
> This makes sense, since the test compiler is not required to be an
> in-project script,

Which is not a problem.  I am very happy that my dependencies
include headers from /usr and not just those in the package.

> and not even to be specified as an absolute path
> -- it can be a command like "sh" or "/usr/bin/env perl".

which is one, indeed.

>> - it would be equally useful to be able to add dependencies
>>  to a category of test drivers.  Say, with the documented
>>  example:
>> 
>>    TESTS = foo.pl bar.py baz
>>    TEST_EXTENSIONS = .pl .py
>>    PL_LOG_COMPILER = $(PERL)
>>    AM_PL_LOG_FLAGS = -w
>>    PY_LOG_COMPILER = $(PYTHON)
>>    AM_PY_LOG_FLAGS = -v
>>    LOG_COMPILER = ./wrapper-script
>>    AM_LOG_FLAGS = -d
>> 
>> be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>> LOG_DEPENDENCIES.
>> 
> This seems like a cool feature, but not trivial to implement in the current
> Automake code base.  Still, I have some half-finished patches to rework
> the parallel-tests harness in Automake-NG (reviews welcome, BTW ;-),

I'll pull :)

> and
> once those are in place, implementing your proposed feature in Automake-NG
> might be very easy.

Good news.