Re: Testing groff. Was: GNUism in groff tests

[John Gardner]

> and they don't have to be unit tests.  In some ways, that's the worst
> option because they tend to be trivial functions being tested

Nobody's saying they should *all* be unit-tests. It's better to test
components in isolation first, verify they do what they're supposed to,
then follow up with more elaborate tests. That way, a regression is easier
to spot and narrow down when a test failure occurs. BDD-style testing can
help — I find tests easier to follow when they're written like plain English
<https://travis-ci.org/file-icons/atom/jobs/627541381#L506>:


Tree-view
  When a file entry is displayed
    When it matches an icon
      ✓ it shows the icon next to its name
      ✓ it removes the built-in icon class

Archive-view
  When a zip-file is opened
    ✓ it shows icons for each of its contents
    When a file is displayed
      ✓ it shows an icon next to its filename
      ✓ it tests path-sensitive rules


Once the simple stuff is covered, you can move on to running pixel-diffs
against corpora of documents (which don't necessarily have to use a
BDD-flavoured approach to testing).

I'm something of a pedantic prick when it comes to airtight test coverage,
so a lot of this mail probably stems from personal bias.



On Mon, 6 Jan 2020 at 05:20, Ralph Corderoy <address@hidden> wrote:

> Hi Ingo,
>
> > Werner wrote:
> > > I think the proper way for testing groff would be to make it run
> > > with a fuzzer,
> >
> > Yes, that would no doubt be useful.
> ...
> > is kind of orthogonal to developing a test suite, though.
> ...
> > A test suite, on the other hand, is most useful for making sure no
> > regressions creep into the functionality,
>
> I think that's groff's pressing need: stopping regressions.  Given what
> I imagine is the part-time nature of contributions, to a large code
> base, that implements a complex system of parts, e.g. the troff
> language, I suspect a fix to more delicate bug may be quite high risk of
> introducing a regression, and often this will be only apparent in the
> PostScript, PDF, or pixels on the page being different; no warnings,
> errors, or dumped core.
>
> That bug I referred to earlier is a good example.
>
>     https://bugs.launchpad.net/ubuntu/+source/groff/+bug/42764
>
>     The footer printed with the -mm macros has changed position between
>     groff versions 1.17.2 and 1.18
>     ...
>         .\" groff -mm -ma4 footermoves.tr >footermoves.ps
>         .\"
>         .PGFORM 21c-2i 29.7c-1i 1i 1
>         .PH "'hl'hc'hr'"
>         .PF "'fl'fc'fr'"
>         .\"
>         .S +1
>         .P
>         Footer not in a constant place across groff versions.
>
>     If the `.S +1' is commented out the problem doesn't occur.
>
> Fixing known bugs, finding more bugs with fuzzing, implementing new
> features, or adding something from another troff, are all less fun to do
> if one's constantly concerned that there's not a good safety net to stop
> faults getting released.  And a buggy groff version or two means
> documents have to start testing the troff implementation and its version
> to implement workarounds.
>
> So I think regression tests come first, and they don't have to be unit
> tests.  In some ways, that's the worst option because they tend to be
> trivial functions being tested; it's the interplay of the whole system
> that matters.  A corpus of documents and golden eyeballed output, say
> ditroff, PostScript, or PNG to suit, would be a good start.  Even if
> it's just a script that compares renderings of all the documents shipped
> with groff between two built-but-not-installed groffs.
>
> Do any of the other troffs have regression tests to our benefit?
>
> --
> Cheers, Ralph.
>
>