Re: Treating tests as special case

[Pjotr Prins]

On Thu, Apr 05, 2018 at 08:21:15AM +0200, Björn Höfling wrote:
> great ideas!
> 
> Last night I did a 
> 
> guix pull && guix package -i git
> 
> We have substitutes, right? Yeah, but someone updated git, on my new
> machine I didn't configure berlin.guixsd.org yet and hydra didn't have
> any substitutes (build wasn't started yet?).
> 
> Building git was relatively fast, but all the tests took ages. And it
> was just git. It should work. The git maintainers ran the tests. Marius
> when he updated it in commit 5c151862c ran the tests. And that should
> be enough of testing. Let's skip the tests.

Not exactly what I am proposing ;). But, even so, I think we should
have a switch for turning off tests. Let the builder decide what is
good or bad. Too much nannying serves no one.

> On the other hand, if I create a new package definition and forget to
> run the tests. If upstream is too sloppy, did not run the tests and had
> no continuous integration. Who will run the tests then?

Hydra should always test before providing a hash that testing is done.

> What if I build my package with different sources?
> 
> And you mentioned different environment conditions like machine and
> kernel. We still have "only" 70-90% reproducibility. The complement
> should have tests enabled. And the question "is my package
> reproducible?" is not trivial to answer, and is not computable.

Well, I believe that case is overrated and we prove that by actually
providing binary substitutes without testing ;)

> We saw tests that failed only in 2% of the runs and were fine in 98%.
> If we would run those tests "just once", we couldn't figure out that
> there is a problem (assuming the problem really is in the software, not
> just the tests).
> 
> There could also be practible problems with that: If all write there
> software nice and with autoconfigure and we just have a "make && make
> test && make install" it's easy to skip the test. But for more
> complicated things we have to find a way to tell the build-system how
> to skip tests.

Totally agree. At this point I patch the tree not to run tests.

Pj.