Re: faster tests [was: rhel8 test failure confirmation?]

[Bogdan]

Bogdan <bogdro_rep@gmx.us>, Tue Apr 04 2023 21:51:52 GMT+0200 (Central 
European Summer Time)
> Karl Berry <karl@freefriends.org>, Mon Apr 03 2023 02:08:22 GMT+0200 
> (Central European Summer Time)
> >      # A trick to make the test run muuuch faster, by avoiding repeated
> >      # runs of aclocal (one order of magnitude improvement in speed!).
> >      echo 'AC_INIT(x,0) AM_INIT_AUTOMAKE' > configure.ac
> >
> > Hadn't noticed this before. Maybe you could see what tests are 
> > currently
> > taking the longest to run, and see if the above helps speed them up?
>
>
>   Sure, when I eventually find the time to run 'time' :). I was hoping 
> to have some results by now, hence the delay.
>   Is there a way to time individual tests, or do I have to run 'time 
> make check TESTS=...'?
>
>
> > It seems a somewhat weird thing to do, but if it saves enough time, I
> > guess it would be worth it. --thanks, karl.
>
>
>   Yes, it is a bit weird and I don't know if this can negatively 
> impact the tests themselves in functionalities which may rely on 
> aclocal.m4 to be correct. Certainly the tests that add or remove macro 
> calls to or from configure.ac need to have aclocal.m4 up-to-date and 
> so they need to run aclocal after each modification, as I see it.


 Coming back to the subject: I chose tests which:

- run 'aclocal' in a loop (assuming it is indented):
        egrep '^[ \t]+\$ACLOCAL' t/*sh

- run 'aclocal' without arguments more than once:
        grep -c '\$ACLOCAL[ \t]*$' t/*.sh | /bin/grep -v ':[01]$'

Then, I analysed the files and added the trick from t/backcompat2.sh 
(if possible) and/or removed the extra calls to $ACLOCAL (if possible).

Results of the analysis or gain in time:

t/aclocal-path-install-serial.sh: cannot change here, aclocal is under 
test

t/backcompat-acout.sh: 35 -> 24s

t/deprecated-acinit.sh: cannot change here, aclocal is under test

t/dist-auxdir-many-subdirs.sh: 1:23 -> 1:18

t/dist-auxfile.sh: 16 -> 15s

t/dist-no-built-sources.sh: 8s, cannot change here (uses current 
configure.ac)

t/suffix.sh: 8s, cannot change here (uses current configure.ac)

t/test-driver-is-distributed.sh: 16 -> 15s

t/am-prog-cc-c-o.sh: 17 -> 15s, cannot change here (uses current 
configure.ac)

t/ar-lib3.sh: cannot change here (uses current configure.ac)

t/ar-lib4.sh: cannot change here (uses current configure.ac)

t/asm2.sh: 14 -> 12s

t/asm3.sh: 14 -> 12s

t/asm.sh: 12 -> 10s

t/backcompat3.sh: 12 -> 10s

t/conffile-leading-dot.sh: 10 -> 9s

t/confh.sh: 9 -> 8s

t/copy.sh: cannot change here (uses current configure.ac)

t/dir-named-obj-is-bad.sh: cannot change here (uses current configure.ac)

t/distcheck-configure-flags-subpkg.sh: 10 -> 9s

t/distcheck-hook2.sh: 9.8 -> 9.2s

t/dist-shar.sh: 8 -> 7-8s

t/dist-tarZ.sh: skipped

t/fn99subdir.sh: 12 -> 11s

t/fort2.sh: 8-9 -> 7.5s

t/libobj16a.sh: 29-34 -> 28s

t/libobj16b.sh: 29 -> 28s

t/libobj19.sh: 22 -> 21s

t/libobj-basic.sh: 20 -> 20s

t/lisp2.sh: 9 -> 7s

t/nodef2.sh: 6 -> 5s

t/nodef.sh: 5 -> 4s

t/objc-basic.sh: cannot change here (changes configure.ac afterwards)

t/objcxx-basic.sh: cannot change here (changes configure.ac afterwards)

t/pr401b.sh: 1:42-43 -> 1:38

t/pr401c.sh: 50 -> 48s

t/pr401.sh: 41 -> 38s

t/serial-tests.sh: 6 -> 5s

t/silent-configsite.sh: cannot change here (changes configure.ac 
afterwards)

t/strictness-precedence.sh: 12 -> 9s

t/subobj6.sh: 10-13 -> 10s

t/subpkg2.sh: cannot change here, aclocal is under test

t/subpkg3.sh: cannot change here, aclocal may be under test

t/subpkg4.sh: cannot change here, aclocal may be under test

t/subpkg.sh: cannot change here, aclocal is under test

t/subpkg-yacc.sh: cannot change here, aclocal may be under test

t/vala-configure.sh: 22 -> 21s

t/vala-non-recursive-setup.sh: skipped

t/warning-groups-win-over-strictness.sh: 7 -> 5s

t/warnings-precedence.sh: 11-12 -> 9s

t/warnings-strictness-interactions.sh: 4-5 -> 4s

t/warnopts.sh: 5 -> 4s

Short version: after a few hours of testing and modifications, I *may* 
have saved up to 1 minute and 12 seconds of testing...

You may look at the attached patch as a result of the investigation 
and then ... you're free to completely ignore it :). It works for me, 
but I wonder if it won't cause more confusion than it's worth...

Funny thing - removing $ACLOCAL from t/backcompat.sh actually makes it 
run much SLOWER - from 30s to 55s. No idea why.

Other conclusions:

- having 1277 .sh files in 't/' means that even if each runs in 30 
seconds, you have 10 hours of testing just from the number of tests,

- it may be better to determine if there are duplicate tests (but with 
different names, circumstances, etc.) and eliminate those or merge 
tests which are "close enough" each other (like one being just another 
test case for the other) - this may be a significant gain,

- as you see above, t/pr401b.sh takes 1m42s to run. I wonder if e.g. 
running the 'distcheck' target in tests would be the main factor of 
the running time. Seems to do a bit of work -  reconfigure, package, 
run tests, install, uninstall, verify, ... Same case for t/pr401c.sh 
and t/pr401.sh, although shorter times. Sure, it *is* needed 
sometimes, but maybe a thing worth looking at if it's needed 
everywhere it's used. Or leave it, just in case, or make the target 
actually run faster...

--
Regards - Bogdan ('bogdro') D.                 (GNU/Linux & FreeDOS)
X86 assembly (DOS, GNU/Linux):    http://bogdro.evai.pl/index-en.php
Soft(EN): http://bogdro.evai.pl/soft  http://bogdro.evai.pl/soft4asm
www.Xiph.org  www.TorProject.org  www.LibreOffice.org  www.GnuPG.org

automake-skip-aclocal-in-tests-mail.diff
Description: Text Data