Re: Treating tests as special case

[Björn Höfling]

On Thu, 5 Apr 2018 07:24:39 +0200
Pjotr Prins <address@hidden> wrote:

> Last night I was watching Rich Hickey's on Specs and deployment. It is
> a very interesting talk in many ways, recommended. He talks about
> tests at 1:02 into the talk:
> 
>   https://www.youtube.com/watch?v=oyLBGkS5ICk
> 
> and he gave me a new insight which rang immediately true. He said:
> what is the point of running tests everywhere? If two people test the
> same thing, what is the added value of that? (I paraphrase)
> 
> With Guix a reproducibly building package generates the same Hash on
> all dependencies. Running the same tests every time on that makes no
> sense.
> 
> And this hooks in with my main peeve about building from source. The
> building takes long enough. Testing takes incredibly long with many
> packages (especially language related) and are usually single core
> (unlike the build). It is also bad for our carbon foot print. Assuming
> everyone uses Guix on the planet, is that where we want to end up?
> 
> Burning down the house.
> 
> Like we pull substitutes we could pull a list of hashes of test cases
> that are known to work (on Hydra or elsewhere). This is much lighter
> than storing substitutes, so when the binaries get removed we can
> still retain the test hashes and have fast builds. Also true for guix
> repo itself.
> 
> I know there are two 'inputs' I am not accounting for: (1) hardware
> variants and (2) the Linux kernel. But, honestly, I do not think we
> are in the business of testing those. We can assume these work. If
> not, any issues will be found in other ways (typically a segfault ;).
> Our tests are generally meaningless when it comes to (1) and (2). And
> packages that build differently on different platforms, like openblas,
> we should opt out on. 
> 
> I think this would be a cool innovation (in more ways than one).
> 
> Pj.

Hi Pjotr,

great ideas!

Last night I did a 

guix pull && guix package -i git

We have substitutes, right? Yeah, but someone updated git, on my new
machine I didn't configure berlin.guixsd.org yet and hydra didn't have
any substitutes (build wasn't started yet?).

Building git was relatively fast, but all the tests took ages. And it
was just git. It should work. The git maintainers ran the tests. Marius
when he updated it in commit 5c151862c ran the tests. And that should
be enough of testing. Let's skip the tests.

On the other hand, if I create a new package definition and forget to
run the tests. If upstream is too sloppy, did not run the tests and had
no continuous integration. Who will run the tests then?

What if I build my package with different sources?

And you mentioned different environment conditions like machine and
kernel. We still have "only" 70-90% reproducibility. The complement
should have tests enabled. And the question "is my package
reproducible?" is not trivial to answer, and is not computable.

We saw tests that failed only in 2% of the runs and were fine in 98%.
If we would run those tests "just once", we couldn't figure out that
there is a problem (assuming the problem really is in the software, not
just the tests).

There could also be practible problems with that: If all write there
software nice and with autoconfigure and we just have a "make && make
test && make install" it's easy to skip the test. But for more
complicated things we have to find a way to tell the build-system how
to skip tests.

Björn

pgpeILl30y5G4.pgp
Description: OpenPGP digital signature