Re: Excluding FAILs from UNSUPPORTED test cases

[Jacob Bachmeyer]

Arsen Arsenović wrote:
> Hi,
>
> On Sunday, 25 September 2022 06:24:06 CEST you wrote:
>   
> > Running ${tool}-dg-prune earlier works both ways, allowing the
> > callback prune routine to also remove or modify any output that might
> > confuse the message-matching mechanism.  Really, given the documented
> > (such as it is) feature for ${tool}-dg-prune to be able to declare
> > the entire test unsupported, I would expect it to filter the results
> > before messages are checked.
> >     
> Re-reading the docs, I think you're right in terms of the semantics that 
> the ${tool}-dg-prune callback should implement.
>
> As a test, I re-ran the GCC test suite after moving the invocation of   
> ${tool}-dg-prune output to be right after ${tool}-dg-test (so, line 700 
> of lib/dg.exp), which introduced 14361 new FAILs (but also fixed a few 
> tests).

The tests that it fixed probably need some attention in the form of 
moving some "preclean" steps to GCC's ${tool}-dg-test.

>   It'd appear that all but sixteen of these new fails are a 
> result of the GCC-specific dg-message and multiline.exp, and some of 
> those sixteen fail due to trying to test for the presence of strings 
> they prune (i.e. they want to detect that there's at least one of some 
> diagnostic, AFAICT).  I'll see if I can do anything to remedy these.
>   

Looking at this again, I now agree with your earlier position on the 
timing of ${tool}-dg-prune, since ${tool}-dg-test can perform any needed 
immediate post-processing itself before returning the results.

Keeping the current interface while fixing the {PASS,FAIL}+UNSUPPORTED 
issue is also possible:  we can build up a list of results from scanning 
the messages, but not actually report them to the framework until after 
checking the return value of ${tool}-dg-prune for an "abandon test" 
flag.  Note that ${tool}-dg-test could produce this flag in lieu of 
actual output and ${tool}-dg-prune simply pass it along.  (I suspect 
that the current Tcl list facilities that would enable this were not yet 
available when dg.exp was written more than 20 years ago.)

The main reason I am reluctant to simply accept the "abandon test" flags 
directly from ${tool}-dg-test is that that would completely break 
backwards compatibility:  testsuites depending on this would absolutely 
need DejaGnu 1.6.4, and I think we may be able to get most of the way 
there (at least getting an UNSUPPORTED result reliably into the summary) 
with existing code.

> In the meanwhile, I filed a bug so that this issue can be tracked in 
> the GNU bug tracker.
>   
Excellent!  It is now tracked as PR58065.  I expect to have a regression 
test and a preliminary patch in a few days.


-- Jacob