Re: GNUism in groff tests, was: pic anomalies

[Larry McVoy]

I can also offer up BitKeeper's test harness, it's super simple but worked
really well for us.

Here is an example test:

echo $N Demonstrate symlink chain to bk bin .........................$NL
if [ "$PLATFORM" = "WIN32" ]
then    echo skipped
        exit 0
fi

mkdir -p a/bin
mkdir -p b/bin
ln -s `bk bin`/bk a/bin/bk
ln -s "$HERE"/a/bin/bk b/bin/bk

test "`bk bin`" = "`b/bin/bk bin`" || fail
echo OK

The tests are all (portable) shell scripts ($N is -n and $NL is \c, the
setup script figures out which sort of sh you have and sets those).
It's a lot like the original perl test suite, it prints stuff like

Demonstrate symlink chain to bk bin .........................OK

or 

Demonstrate symlink chain to bk bin .........................failed

Each test is a "t.file" like "t.test-this" and "t.test-that".

The harness has been evolved to run all the tests in parallel.

It's slick.

On Tue, Dec 31, 2019 at 01:13:19PM +1100, John Gardner wrote:
> > As long as these tests use bash(1), i'm very reluctant to do that,
> > even though in general, running a test suite certainly makes sense
> > before you commit a package update.  Given the so far very small
> > test coverage, the tests don't help much for package testing yet.
> > Then again, that is likely to improve in the future, so having them
> > portable would be nice...
> 
> Wouldn't having a TAP <https://testanything.org/> harness be preferable to
> hand-spun shell-scripts? Groff already depends on Perl, so it's not as
> though using prove(1) to run tests would add much weight.
> 
> I was rather shocked to learn such a widely-used program as Groff had such
> minimal test coverage. Testing the output of binary formats is
> understandably difficult, but I think most of those pains can be alleviated
> by testing against Groff's intermediate output format first, and then
> having unit tests assert proper transformation after.
> 
> 
> On Tue, 31 Dec 2019 at 03:14, Ingo Schwarze <address@hidden> wrote:
> 
> > Hi,
> >
> > Colin Watson wrote on Mon, Dec 30, 2019 at 01:39:08PM +0000:
> > > On Mon, Dec 30, 2019 at 01:39:38PM +0100, Ingo Schwarze wrote:
> >
> > >> For example, right now, after "git pull" and building from source,
> > >> nine out of the fourteen tests fail for me on OpenBSD-current,
> >
> >   schwarze@isnote $ make check
> >   [...]
> >   PASS: contrib/gdiffmk/tests/gdiffmk_tests.sh
> >   PASS: contrib/hdtbl/examples/test-hdtbl.sh
> >   PASS: contrib/mom/examples/tests-mom.sh
> >   PASS: src/roff/groff/tests/regression-56555.sh
> >   FAIL: src/roff/groff/tests/on-latin1-device-oq-is-0x27.sh
> >   FAIL: src/roff/groff/tests/string_case_xform_requests.sh
> >   XFAIL: src/roff/groff/tests/string_case_xform_unicode_escape.sh
> >   FAIL: src/roff/groff/tests/string_case_xform_errors.sh
> >   FAIL: tmac/tests/an-old_CS_register_off.sh
> >   FAIL: tmac/tests/an-old_CS_register_on.sh
> >   FAIL: tmac/tests/an-old_CS_register_unspecified.sh
> >   FAIL: tmac/tests/an-old_CT_register_off.sh
> >   FAIL: tmac/tests/an-old_CT_register_on.sh
> >   FAIL: tmac/tests/an-old_CT_register_unspecified.sh
> >
> > I had a closer look and it turns out all nine failing tests start
> > with the line
> >
> >   #!/usr/bin/env bash
> >
> > I don't have bash(1) installed on my machines and i don't want it
> > installed there.  One of the reasons is that some autoconfiguration
> > systems and some application softwares look for bash and run it if
> > it is installed.
> >
> > In general, i only install GNU versions of POSIX tools with a "g"
> > prefix (e.g. /usr/local/bin/gmake, /usr/local/bin/gtar) to make
> > it less likely that something picks them up, but even so, having
> > them installed is a risk, so i avoid even that as much as i can.
> >
> > I just temporarily installed bash(1) for testing purposes to make sure
> > that's currently the only problem, and it is: with bash(1) installed,
> > all 14 tests currently succeed.
> >
> > But i deleted /usr/local/bin/bash again, right away.
> >
> > > FWIW, I run these tests as part of the Debian package build and enforce
> > > that they pass.  This is obviously only of limited use since it's only
> > > after releases, but it's better than nothing.  (I would hope that
> > > maintainers also run "make check" prior to release, perhaps via "make
> > > distcheck", although I don't actually know.)
> >
> > As long as these tests use bash(1), i'm very reluctant to do that,
> > even though in general, running a test suite certainly makes sense
> > before you commit a package update.  Given the so far very small
> > test coverage, the tests don't help much for package testing yet.
> > Then again, that is likely to improve in the future, so having them
> > portable would be nice...
> >
> > Yours,
> >   Ingo
> >
> >

-- 
---
Larry McVoy                  lm at mcvoy.com             
http://www.mcvoy.com/lm