Re: [PATCH DejaGNU/GCC 0/1] Support per-test execution timeout factor

[Hans-Peter Nilsson]

On Wed, 3 Jan 2024, Maciej W. Rozycki wrote:

> On Wed, 3 Jan 2024, Hans-Peter Nilsson wrote:
> 
> > >  The test execution timeout is different from the tool execution timeout 
> > > where it is GCC execution that is being guarded against taking excessive 
> > > amount of time on the test host rather than the resulting test case 
> > > executable run on the target afterwards, as concerned here.  GCC already 
> > > has a `dg-timeout-factor' setting for the tool execution timeout, but has 
> > > no means to increase the test execution timeout.  The GCC side of these 
> > > changes adds a corresponding `dg-test-timeout-factor' setting.
> > 
> > Hmm.  I think it would be more correct to emphasize that the 
> > existing dg-timeout-factor affects both the tool execution *and* 
> > the test execution, whereas your new dg-test-timeout-factor only 
> > affects the test execution.  (And still measured on the host.)
> 
>  Not really, `dg-timeout-factor' is only applied to tool execution and it 
> doesn't affect test execution.

Let's stop here: that statement is just incorrect.

There might be a use for separating timeouts, but the premise of 
dg-timeout-factor not affecting the execution of an (executable) 
test is plain wrong.  Something is off here.  Are we using the 
same terminology?

Please revisit the setup where the patch made a difference and 
report it for others to repeat, something like the following:
(Beware of typos, I didn't copy-paste like I usually do.)

A recent observation is me testing MMIX, where gcc+newlib is 
build and with --prefix and $PATH pointing at pre-installed 
binutils and simulator.  I test it all with "make -k check 
--target_board=mmixware-sim".  (This should all be familiar; 
pick another target and baseboard or use a native+unix.exp if 
you prefer.  Also JFTR I usually do it by means of 
contrib/regression/btest-gcc.sh - except of course when 
inspecting and manual testing.)

Anyway a repeatable case where dg-timeout-factor then makes a 
difference for the timeout is for libstdc++-v3 test-case 
20_util/hash/quality.cc.  I recently committed a patch adding 
dg-timeout-factor 3 for that test (26fe2808d8).  Let's consider 
the situation *before* that commit.

For the mmix simulator and the particular host where I ran that 
test, the test normally executes in very close to 6 minutes, and 
as the default timeout of 360 seconds, it sometimes times out 
when the machine is busy.  To make *sure* it times out for case 
of proof here, I edit the -DNTESTS=1 simulator setting to 
-DNTESTS=2.  I execute just this test by for example "make 
check-target-libstdc++-v3 
RUNTESTFLAGS=--target_board=mmixware-sim\ 
conformance.exp=quality.cc" (beware of quoting issues - which 
should be familiar to you).

That NTESTS=2 makes the execution time go up to 13 minutes 
elapsed time and the test gets a "WARNING: program timed out" 
and a failure for that test.  I also see a (timeout = 360) in 
the libstdc++.log - admittedly for the compilation line, but 
the timeout is consistent with being applied to the execution 
as well.

Then, apply the commit, which adds a line with dg-timeout-factor 3
(bah, I had to do it manually because of that edited -DNTESTS=2 line).

After, when I un the same command-line, the test *does not time 
out, it passes* and I see a (timeout = 1080) next to the 
compilation line in the .log - but it's apparently applied to 
the test run as well.

That, as well as numerous previous commits, is consistent with 
dg-timeout-factor affecting the execution time, not just the 
compilation time.

Of course, there may be some sub-test-suite that has a bug. I'm 
*guessing* you misinterpret observations that lead up to this 
patch-set, perhaps a bug in some sub-test.exp.

>  Timeout value reporting used to be limited 
> in DejaGNU, but you can enable it easily now by adding the DejaGNU patch 
> series referred in the cover letter and see that `dg-timeout-factor' is 
> ignored for test execution.

Please state a case where I can observe it being ignored.

> > Usually the compilation time is close to 0, so is this based on 
> > an actual need more than an itchy "wart"?
> > 
> > Or did I miss something?
> 
>  Compilation is usually quite fast, but this is not always the case.  If 
> you look at the tests that do use `dg-timeout-factor' in GCC, and some 
> commits that added the setting, then you ought to find actual use cases.  

I've not only looked at such commits, I've done quite a few 
myself.  I'd say most such commits are for test execution, some 
are for compilation.  Did you miss the ones where the commit log 
mentions "slow simulator" or "slow board"?

> I saw at least one such a test that takes an awful lot of time here on a 
> reasonably fast host machine and still passes where GCC has been built 
> with optimisation enabled, but does time out in the compilation phase if 
> the compiler has been built at -O0 for debugging purposes.  I'd have to 
> chase it though if you couldn't find it as I haven't written the name 
> down.

Maybe the one Richard Sandiford mentioned in a reply.  But, 
that's compile only.

>  So yes, `dg-timeout-factor' does have its use, but it is different from 
> that of `dg-test-timeout-factor', hence the need for a separate setting.

No.  They overlap: dg-timeout-factor is for both.

(If you want to *remove* that overlap, please make sure you 
migrate the right subset to use dg-test-timeout-factor.)

brgds, H-P