Re: GNUism in groff tests, was: pic anomalies

[John Gardner]

> As long as these tests use bash(1), i'm very reluctant to do that,
> even though in general, running a test suite certainly makes sense
> before you commit a package update.  Given the so far very small
> test coverage, the tests don't help much for package testing yet.
> Then again, that is likely to improve in the future, so having them
> portable would be nice...

Wouldn't having a TAP <https://testanything.org/> harness be preferable to
hand-spun shell-scripts? Groff already depends on Perl, so it's not as
though using prove(1) to run tests would add much weight.

I was rather shocked to learn such a widely-used program as Groff had such
minimal test coverage. Testing the output of binary formats is
understandably difficult, but I think most of those pains can be alleviated
by testing against Groff's intermediate output format first, and then
having unit tests assert proper transformation after.


On Tue, 31 Dec 2019 at 03:14, Ingo Schwarze <address@hidden> wrote:

> Hi,
>
> Colin Watson wrote on Mon, Dec 30, 2019 at 01:39:08PM +0000:
> > On Mon, Dec 30, 2019 at 01:39:38PM +0100, Ingo Schwarze wrote:
>
> >> For example, right now, after "git pull" and building from source,
> >> nine out of the fourteen tests fail for me on OpenBSD-current,
>
>   schwarze@isnote $ make check
>   [...]
>   PASS: contrib/gdiffmk/tests/gdiffmk_tests.sh
>   PASS: contrib/hdtbl/examples/test-hdtbl.sh
>   PASS: contrib/mom/examples/tests-mom.sh
>   PASS: src/roff/groff/tests/regression-56555.sh
>   FAIL: src/roff/groff/tests/on-latin1-device-oq-is-0x27.sh
>   FAIL: src/roff/groff/tests/string_case_xform_requests.sh
>   XFAIL: src/roff/groff/tests/string_case_xform_unicode_escape.sh
>   FAIL: src/roff/groff/tests/string_case_xform_errors.sh
>   FAIL: tmac/tests/an-old_CS_register_off.sh
>   FAIL: tmac/tests/an-old_CS_register_on.sh
>   FAIL: tmac/tests/an-old_CS_register_unspecified.sh
>   FAIL: tmac/tests/an-old_CT_register_off.sh
>   FAIL: tmac/tests/an-old_CT_register_on.sh
>   FAIL: tmac/tests/an-old_CT_register_unspecified.sh
>
> I had a closer look and it turns out all nine failing tests start
> with the line
>
>   #!/usr/bin/env bash
>
> I don't have bash(1) installed on my machines and i don't want it
> installed there.  One of the reasons is that some autoconfiguration
> systems and some application softwares look for bash and run it if
> it is installed.
>
> In general, i only install GNU versions of POSIX tools with a "g"
> prefix (e.g. /usr/local/bin/gmake, /usr/local/bin/gtar) to make
> it less likely that something picks them up, but even so, having
> them installed is a risk, so i avoid even that as much as i can.
>
> I just temporarily installed bash(1) for testing purposes to make sure
> that's currently the only problem, and it is: with bash(1) installed,
> all 14 tests currently succeed.
>
> But i deleted /usr/local/bin/bash again, right away.
>
> > FWIW, I run these tests as part of the Debian package build and enforce
> > that they pass.  This is obviously only of limited use since it's only
> > after releases, but it's better than nothing.  (I would hope that
> > maintainers also run "make check" prior to release, perhaps via "make
> > distcheck", although I don't actually know.)
>
> As long as these tests use bash(1), i'm very reluctant to do that,
> even though in general, running a test suite certainly makes sense
> before you commit a package update.  Given the so far very small
> test coverage, the tests don't help much for package testing yet.
> Then again, that is likely to improve in the future, so having them
> portable would be nice...
>
> Yours,
>   Ingo
>
>