Re: How to check installed binaries using the same tests in the source code?

[Bruno Haible]

Santiago Vila wrote:
> > For our Debian CI framework (ci.debian.net)
> > we are adding tests to every package, called
> > autopkgtests.
> > 
> > The gettext package already has a "make check"
> > target, but it checks the just-built package,
> > not the installed version in /usr which
> > is what we would need for the CI framework.
...
> > (I see there is an "installcheck" target

Yes, for GNU packages the GCS specify that this is the job of
'make installcheck'. [1]

There are two questions:
  1) What should an installcheck do?
  2) How should it be implemented? As a Makefile target, or as
     a separate program?

> > which apparently
> > checks for --version and --help but I think nothing else)

A while ago, 'installcheck' was even a no-op for most packages [2].

What should 'installcheck' reasonably check (assuming "make check"
was already run before "make install")?

(A) For packages that install independent programs, like coreutils
    or diffutils, the most frequent types of installation bugs are:
      - program not installed or installed in wrong location,
      - shared libraries that are needed by the programs are not
        not installed or installed in a location not listed in RPATH.
    The Automake-generated default 'installcheck', that — as you say —
    invokes all programs with --version and --help is a good and fast
    test against these installation bugs.

(B) For packages that install programs that invoke each other or that
    rely on installed data files, some of the test suite of "make check"
    ought to be performed by "make installcheck". For example, in gettext,
      - msgfmt and xgettext rely on some data files for handling XML
        files. So, these tests ought to be performed by "make installcheck".
      - Similarly, msginit invokes some auxiliary programs. Therefore,
        the msginit tests ought to be done by "make installcheck" as well.

(C) For packages that install libraries, "make installcheck" should
      - verify that every include file that ought to be installed actually
        is installed and includable,
      - verify that each function and variable that ought to be defined
        in the library actually is (and is not, for example, hidden through
        some visibility control).
      - optionally, verify that the library does not defined other symbols
        without a certain prefix.
    For this, it seems useful to run the entire test suite of the library
    against the installed library.

> > Somebody suggested the trick of symlinking the
> > executables to their /usr-installed counterpart
> > before running make check

This is a nice trick for doing the tests in case (A).

For case (B), this is not fool-proof, because it could rely on built artifacts
in the build directory. And for case (C), it is not applicable at all.

So, for case (B), I think a better approach is
  1) to do a "make clean" first (i.e. 'installcheck' depends on 'clean'),
  2) to use a different 'init-env' (or an 'init-env' with conditional code),
     with a subset of the tests that "make check" runs.

Simon Josefsson wrote:
> For what’s its worth, I added support for this in recent libidn releases so 
> that distribution maintainers can run tests/standalone.sh in an unpacked (but 
> not built) tarball and it will run the tests-suite on the system-installed 
> package.  I think this is a good pattern: running “make check” is usually 
> heavily tied into the build infrastructure. But the self tests in upstream 
> are often what you want to run on the system-installed package to make sure 
> it works. There is no standard way to do this in Autotools. Sometimes 
> standalone testing catches problems that “make check” fail to catch, for 
> things related to the installation (or just random reasons).
> 
> https://salsa.debian.org/debian/libidn/-/blob/master/NEWS?ref_type=heads#L19

What I like about this approach is
  - It can run all tests without code duplication.
  - It runs the installchecks when no built artifacts are present.
What I dislike:
  - It needs a way to pass the compiler and compiler options. Duplicate
    the way 'configure' does it, via $CC and $CFLAGS?
  - It needs more work, until it produces a test-suite.log like Automake does.
  - It's not the way the GNU Coding Standards specify it.

How to proceed?

Would it make sense that I add 'installcheck' to packages
  - of type (A): gperf, diffutils,
  - of type (C): libunistring, libsigsegv,
  - of type (B): gettext ?

Would Debian's CI framework then do

  ./configure [OPTIONS]
  make installcheck

when possible?

Bruno

[1] https://www.gnu.org/prep/standards/html_node/Standard-Targets.html
[2] https://lists.gnu.org/archive/html/bug-coreutils/2012-05/msg00053.html