Re: Treating tests as special case

[Mark H Weaver]

Hi Pjotr,

Pjotr Prins <address@hidden> writes:

> and he gave me a new insight which rang immediately true. He said:
> what is the point of running tests everywhere? If two people test the
> same thing, what is the added value of that? (I paraphrase)
>
> With Guix a reproducibly building package generates the same Hash on
> all dependencies. Running the same tests every time on that makes no
> sense.

I appreciate your thoughts on this, but I respectfully disagree.

> I know there are two 'inputs' I am not accounting for: (1) hardware
> variants and (2) the Linux kernel. But, honestly, I do not think we
> are in the business of testing those. We can assume these work.

No, we can't.  For example, I recently discovered that GNU Tar fails one
of its tests on my GuixSD system based on Btrfs.  It turned out to be a
real bug in GNU Tar that could lead to data loss when creating an
archive of recently written files, with --sparse enabled.  I fixed it in
commit 45413064c9db1712c845e5a1065aa81f66667abe on core-updates.

I would not have discovered this bug if I had simply assumed that since
GNU Tar passes its tests on ext4fs, it surely must also pass its tests
on every other file system.

> If not, any issues will be found in other ways (typically a segfault
> ;).

The GNU Tar bug on Btrfs would never produce a segfault.  The only way
the bug could be observed is by noticing that data was lost.  I don't
think that's a good way to discover a bug.  I'd much rather discover the
bug by a failing test suite.

Tests on different hardware/kernel/kernel-config/file-system
combinations are quite useful for those who care about reliability of
their systems.  I, for one, would like to keep running test suites on my
own systems.

     Regards,
       Mark