[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Gripes about Grep test suite
From: |
Eli Zaretskii |
Subject: |
Gripes about Grep test suite |
Date: |
Tue, 20 Dec 2011 20:16:53 +0200 |
I'm not sure this warrants a Savannah bug report, but if tell me it
does, I will submit one.
During the past few days I needed to investigate several failures of
Grep 2.9 and 2.10 tests on MS-Windows. I must say that I found the
test suite less user-friendly than I'd like, when it comes to digging
up the reason for a failure. Here's why:
. There's no README or other file in the `tests' directory with
instructions on running and re-running the tests.
The first thing one needs to know how to do is run a single test,
but this is not explained anywhere. As you-all know only too well,
wading through the maze of autoconf-generated Makefiles trying to
figure this out is not for the faint at heart, what with half a
dozen levels of indirection in every target. It took me a while to
figure that out. (Later, I found the recipe buried inside HACKING,
but that file is not part of a release tarball, and I worked with
an official release.)
. Another annoyance is that each test script typically runs several
individual test commands, but the diagnostics left on the log files
is not detailed enough to pinpoint the specific command that
failed. For example, Spencer tests leave this laconic message:
Spencer ere test #113 failed
It is entirely not trivial to find out what Grep command was run
and failed, because the shell script which runs all those tests is
deleted, even when there's a failure. I needed to find out how to
produce that script, then generate it manually, and only then I
could see the failed command.
Other tests leave even less information. E.g., if Grep wrote some
error message to stderr, that error message gets written to the log
file with no context information at all, so it is hard to know
which part(s) of the test triggered the error. I found myself
adding "set -x" to various scripts to unlock these mysteries.
May I suggest that the log file shows the full command that failed?
. Yet another problem is that every run of "make check" wipes out the
test-suite.log file. So if you want to consult past results, you
need to save the file under a different name. WIBNI the file name
included some unique string, like the time of the test? this would
avoid wiping out previous results.
. Last, but not least: the test suite requires a `timeout' command
for some of the tests. What it does when this command is not found
is sub-optimal: it skips the tests that require this command. This
means that Grep built on a new platform cannot be fully tested
without also building a fairly recent version of Coreutils.
May I suggest to include in the test suite a shell script that
would emulate this command for the simple uses needed by the tests?
There are a couple of such scripts floating around (I used one of
them), and I think they are good enough for this purpose.
Thank you for listening to my whining.
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Gripes about Grep test suite,
Eli Zaretskii <=