Re: master 262d0c6: Mark some tests as expensive

[Daniel Martín]

Eli Zaretskii <eliz@gnu.org> writes:
>
> Tramp tests need more time because they involve a remote system.
> Moreover, the time taken by each Tramp test depends on the speed of
> the connection, which you cannot know in advance.

If they depend on the connection speed, or if a host is online or
offline, they also introduce non-determinism that perhaps is not
apparent now, but can cause problems when the Tramp codebase (and its
number of tests) scale. For example, if a Tramp test has a 0.1% chance
of failure because of an unrelated network problem, then if the Tramp
test suite reaches a point where 10000 tests are run per day, people
would be investigating 10 test flakes per day. That's a good reason for
people to lose confidence in the Tramp test suite and ignore failures.

One could argue that the Tramp test suite will never reach such a high
number of tests/contributions, and maybe that's fine, but the problem of
scale exists, IMHO.

>
> I don't, not in general.  Artificially making such changes will run a
> risk of missing real problems, because the test runs in an environment
> different from a real one.  This approach can be a good idea in some
> cases, but in general we should try to run each test as close to
> real-life conditions as possible.

I'd say "we should try to run *some* tests as close to real-life
conditions as possible". By abstracting the environment in some tests,
one could potentially test an infinite number of environments and error
conditions, not only what the test happens to run on. We would still
have a few end-to-end tests that check that the program as a whole works
fine, of course. The trade-off is that writing that kind of hermetic
tests takes more time, specially for packages like Tramp.

Having said that, I think that covering Emacs functionality with an
end-to-end test is much better than having no test at all. But I'd
expect some of those end-to-end tests marked as ":expensive-test" to
evolve into smaller, more hermetic ones, as the test suite matures.