Re: faster tests [was: rhel8 test failure confirmation?]

[Karl Berry]

Hi Bogdan,

    Then, I analysed the files and added the trick from t/backcompat2.sh 
    (if possible) and/or removed the extra calls to $ACLOCAL (if possible).

Thanks much for looking into this.

    Short version: after a few hours of testing and modifications, I
    *may* have saved up to 1 minute and 12 seconds of testing...

Well, at least you get kudos for doing all the research :).

    You may look at the attached patch as a result of the investigation 
    and then ... you're free to completely ignore it :). It works for me, 
    but I wonder if it won't cause more confusion than it's worth...

I agree. Not worth the complications.

    t/backcompat-acout.sh: 35 -> 24s

That seems to me like the only one that might be worth applying the
patch for. Quite a bit more savings than anything else in the list.

>>      # A trick to make the test run muuuch faster, by avoiding repeated
>>      # runs of aclocal (one order of magnitude improvement in speed!).
>>      echo 'AC_INIT(x,0) AM_INIT_AUTOMAKE' > configure.ac

Alternatively, I wonder how much this is really saving. Maybe the trick
should not be used anywhere.

    - having 1277 .sh files in 't/' means that even if each runs in 30 
    seconds, you have 10 hours of testing just from the number of tests,

Indeed. The only practical way to run make check is in parallel.  I
discovered that early on :). It still takes painfully long for me
(10-15min at best, on a new and fast machine).

    - it may be better to determine if there are duplicate tests

Sounds awfully hard to do.

My impression is that the (vast?) majority of tests are the direct
result of bug reports. I would not be inclined to tweak, remove, merge,
or change them. Even if two tests are nearly identical, that "nearly"
can make all the difference.

    - as you see above, t/pr401b.sh takes 1m42s to run. I wonder if e.g. 
    running the 'distcheck' target in tests would be the main factor 

Sounds very likely to me. Distcheck is inherently a lengthy process. I
can't imagine how it can be sped up. Although I agree that 1:42 seems
rather long for a trivial package like those in the tests.

    Same case for t/pr401c.sh and t/pr401.sh, although shorter times.

At a glance, I see required='cc libtoolize' in 401b, whereas 401c and
401 only have cc. Testing libtool really is different, and really does
take time. So I'm not sure there's any low-hanging fruit here.

Thanks again for doing all this work,
Karl