[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: master 262d0c6: Mark some tests as expensive
From: |
Eli Zaretskii |
Subject: |
Re: master 262d0c6: Mark some tests as expensive |
Date: |
Sat, 12 Sep 2020 13:38:01 +0300 |
> From: Daniel Martín <mardani29@yahoo.es>
> Cc: Michael Albinus <michael.albinus@gmx.de>, emacs-devel@gnu.org
> Date: Sat, 12 Sep 2020 12:25:17 +0200
>
> Stefan Kangas <stefan@marxist.se> writes:
> >
> > The idea is to avoid that it will take a very long time to run the unit
> > tests as the test suite grows. Arguably a unit test should never take
> > longer than a second, and even that is in the slow end. If we have
> > 10.000 unit tests taking a second each, just do the math of how long it
> > will take to run (even with parallelization). We should not postpone
> > working on this until we are in that situation, IMHO, because by then it
> > will be a major pain.
>
> I agree. We should differentiate between unit tests (they run in under a
> second) and end-to-end tests, which are slower but still valuable
> because they cover whole scenarios that unit tests can't cover.
Tramp tests need more time because they involve a remote system.
Moreover, the time taken by each Tramp test depends on the speed of
the connection, which you cannot know in advance.
So I conclude that the "under one second" criterion is not smart
enough to be applicable to Tramp tests, and maybe to some others as
well.
I also don't understand the more general issue with how long the test
suite runs. While it runs, you can do something useful on a modern
system (or just go for a coffee), so why does it matter? I've seen
test suites that take a very long time to run (e.g., look at Texinfo
or at Guile), and I never felt I was wasting my time by running those
comprehensive test suites.
I think we are exaggerating the importance of the time it takes to run
the tests.
> > I think the solution for important tests is to refactor the code to make
> > them run faster, for example by avoiding I/O or to make timers trigger
> > immediately.
>
> Yes, I think this is a good approach to follow in general.
I don't, not in general. Artificially making such changes will run a
risk of missing real problems, because the test runs in an environment
different from a real one. This approach can be a good idea in some
cases, but in general we should try to run each test as close to
real-life conditions as possible.
- Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/11
- Re: master 262d0c6: Mark some tests as expensive, Stefan Kangas, 2020/09/11
- Re: master 262d0c6: Mark some tests as expensive, Daniel Martín, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive,
Eli Zaretskii <=
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Eli Zaretskii, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Eli Zaretskii, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/13
- Re: master 262d0c6: Mark some tests as expensive, Stefan Monnier, 2020/09/13
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/13
- Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/13
- Re: master 262d0c6: Mark some tests as expensive, Michael Albinus, 2020/09/12
- Re: master 262d0c6: Mark some tests as expensive, Lars Ingebrigtsen, 2020/09/13