Re: master 262d0c6: Mark some tests as expensive

[Daniel Martín]

Stefan Kangas <stefan@marxist.se> writes:
>
> The idea is to avoid that it will take a very long time to run the unit
> tests as the test suite grows.  Arguably a unit test should never take
> longer than a second, and even that is in the slow end.  If we have
> 10.000 unit tests taking a second each, just do the math of how long it
> will take to run (even with parallelization).  We should not postpone
> working on this until we are in that situation, IMHO, because by then it
> will be a major pain.

I agree. We should differentiate between unit tests (they run in under a
second) and end-to-end tests, which are slower but still valuable
because they cover whole scenarios that unit tests can't cover. To keep
things scalable, we should strive to have plenty of fast unit tests and
fewer end-to-end tests. See
https://martinfowler.com/bliki/TestPyramid.html

>
> I think the solution for important tests is to refactor the code to make
> them run faster, for example by avoiding I/O or to make timers trigger
> immediately.

Yes, I think this is a good approach to follow in general. Many of the
test flakiness in the Emacs suite happen because many tests depend
implicitly in things outside the module they are testing (like
configuration, OS, etc.). Having more stable tests would help us detect
real regressions more accurately.

>
> Best regards,
> Stefan Kangas