Re: master 262d0c6: Mark some tests as expensive

[Michael Albinus]

Stefan Kangas <stefan@marxist.se> writes:

> Hi Michael,

Hi Stefan,

>> Could you please give a reasoning? Declaring tests as expansive decreases
>> heavily their application, giving us less chances to detect errors.
>
> These tests all took 5-60 seconds to run, most in the lower end
> admittedly.

With your change, you have tagged *all* autorevert tests as
:expensive-test. In practice this means, they won't run by default ever,
because people call "make check" only. Any error there, which is not
related to GNU/Linux, will be hidden (hydra and emba run "make
check-expensive", but only for GNU/Linux).

I don't think this is your intention.

> The idea is to avoid that it will take a very long time to run the unit
> tests as the test suite grows.  Arguably a unit test should never take
> longer than a second, and even that is in the slow end.  If we have
> 10.000 unit tests taking a second each, just do the math of how long it
> will take to run (even with parallelization).  We should not postpone
> working on this until we are in that situation, IMHO, because by then it
> will be a major pain.

There have been discussions about how long tests shall run. But there
hasn't been ever a conclusion to limit them to 1 second. Please discuss
this first, before making such changes.

> Best regards,
> Stefan Kangas

Best regards, Michael.