Re: How to check installed binaries using the same tests in the source code?

[Santiago Vila]

(sorry for the late reply)

> There are two questions:
>    1) What should an installcheck do?

Ideally, Debian would like "make installcheck" to be
as similar as possible to "make check", but using
everything (or as much as it's possible) from the
actually installed packages (the ones produced by
the Debian source package), not from the recently-built
tree.

For packages of type (A) there is certainly not a big difference.

This is for example what I did very recently in diffutils
after considering several options:

#!/bin/sh
set -e
PROGRAMS="cmp diff diff3 sdiff"
for a in ${PROGRAMS}; do
  ln -sf /usr/bin/${a} src/${a}
done
make -C tests check

Note: The above works because I also used the keyword
"build-needed", which builds the package
(dpkg-buildpackage -uc -us -b) without running the tests.

If there was a way to create the tests infrastructure
without building the whole package, that would be
very useful, as it would reduce test times, and of course
if "make installcheck" checks installed binaries
then we could avoid the symlinks as well.

> Would it make sense that I add 'installcheck' to packages
>    - of type (A): gperf, diffutils,
>    - of type (C): libunistring, libsigsegv,
>    - of type (B): gettext ?
>
> Would Debian's CI framework then do
>
>    ./configure [OPTIONS]
>    make installcheck
>
> when possible?

Most probably, yes. The idea is to reuse the already existing
tests as much as possible.

So, if you can do that in a way which does not involve
duplication of code (i.e. using essentially the same tests
for "make check" and "make installcheck"), that would be probably
the good way to achieve it.

> (A) For packages that install independent programs, like coreutils
>      or diffutils, the most frequent types of installation bugs are:
>        - program not installed or installed in wrong location,
>        - shared libraries that are needed by the programs are not
>          not installed or installed in a location not listed in RPATH.
>      The Automake-generated default 'installcheck', that — as you say —
>      invokes all programs with --version and --help is a good and fast
>      test against these installation bugs.

I would say that it's even most frequent that some program produces
a different output than the expected.

So, for simple packages like diffutils, the trick above for diffutils
to reuse "make check" is still better than checking --version and --help.

>    2) How should it be implemented? As a Makefile target, or as
>       a separate program?

I think that's up to you. As far as it's easy to use, it does not matter,
so I would chose whatever fits better with GNU coding standards, or with
the package at hand.

[ Note: Maybe I make some more comments about the things I missed, but
I wanted to provide a general answer now instead of delaying my reply
even more ].

Thanks.