Re: verbose testsuite output contains 1M newlines

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Ralf Wildenhues on 11/23/2008 9:37 AM:
> * Eric Blake wrote on Sun, Nov 23, 2008 at 03:56:49PM CET:
>> According to Ralf Wildenhues on 11/23/2008 6:23 AM:
>>> running
>>>   make check-local TESTSUITEFLAGS=-v
>>>
>>> on M4 (git master) produces lots of output on GNU/Linux/bash.
> 
>> The problem stems from the fact that by default, when you
>> specify [stdout] as the third argument to AT_CHECK, autotest emits code
>> that both collects the file stdout (good) and copies the resulting file
>> into the log (annoying), and when -v is in effect, tees that copy to the
>> terminal (bad).  I think the idea behind this decision was that
>> testsuite.log should be self-contained, with everything needed to
>> reproduce why a test failed.
> 
> Yes.  I think the Autotest behavior is more or less ok for most tests.
> Maybe we can have AT_CHECK modifiers or sister macros that allow to
> specify that some output may not be suitable for the screen or the log
> file, due to size or content such as control characters.

Here's my proposal for the autotest side - ignore-nolog, stdout-nolog, and
stderr-nolog.  If you like this, then I can follow up with an m4 patch to
use them on the particular tests where we really don't need to see 1M of
newlines.

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknjOqwACgkQ84KuGfSFAYAq8ACgtpiTKeuuHAg7UXGqc0H6HAxd
DrQAnRzUQksHOi1a9JlvLOmWj4a3Na+V
=l2eS
-----END PGP SIGNATURE-----

>From 5f705e43fce78f68184b7c1e7a2665f52d64c748 Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Mon, 13 Apr 2009 07:10:32 -0600
Subject: [PATCH] Add stdout-nolog and ignore-nolog to AT_CHECK.

* lib/autotest/general.m4 (AT_DIFF_STDERR(stderr-nolog))
(AT_DIFF_STDERR(ignore-nolog), AT_DIFF_STDOUT(stdout-nolog))
(AT_DIFF_STDOUT(ignore-nolog)): New macros.
* tests/autotest.at (Logging): New test.
* doc/autoconf.texi (Writing Testsuites) <AT_CHECK>: Document the
new logging actions.
* NEWS: Likewise.
Reported by Ralf Wildenhues.

Signed-off-by: Eric Blake <address@hidden>
---
 ChangeLog               |   10 ++++++++++
 NEWS                    |    3 ++-
 doc/autoconf.texi       |   23 ++++++++++++++++++++---
 lib/autotest/general.m4 |    6 ++++++
 tests/autotest.at       |   45 +++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 83 insertions(+), 4 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 062198b..d87b650 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,15 @@
 2009-04-13  Eric Blake  <address@hidden>

+       Add stdout-nolog and ignore-nolog to AT_CHECK.
+       * lib/autotest/general.m4 (AT_DIFF_STDERR(stderr-nolog))
+       (AT_DIFF_STDERR(ignore-nolog), AT_DIFF_STDOUT(stdout-nolog))
+       (AT_DIFF_STDOUT(ignore-nolog)): New macros.
+       * tests/autotest.at (Logging): New test.
+       * doc/autoconf.texi (Writing Testsuites) <AT_CHECK>: Document the
+       new logging actions.
+       * NEWS: Likewise.
+       Reported by Ralf Wildenhues.
+
        Teach AT_CHECK about hard failures.
        * lib/autotest/general.m4 (AT_INIT) <at_fn_check_skip>
        <at_fn_check_status, at_fn_group_postprocess>: Handle hard
diff --git a/NEWS b/NEWS
index 854533b..3f537be 100644
--- a/NEWS
+++ b/NEWS
@@ -10,7 +10,8 @@ GNU Autoconf NEWS - User visible changes.
 ** The macro AT_CHECK now understands the concept of hard failure.  If
    a test exits with an unexpected status 99, cleanup actions for the
    test are inhibited and the test is treated as a failure regardless
-   of AT_XFAIL_IF.
+   of AT_XFAIL_IF.  It also understands the new directives
+   ignore-nolog, stdout-nolog, and stderr-nolog.

 ** The autotest macro AT_CHECK_NOESCAPE is now documented.

diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index ff359ad..958e4ef 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -22312,9 +22312,18 @@ Writing Testsuites
 @table @samp
 @item ignore
 The content of the output is ignored, but still captured in the test
-group log (if the test group later fails, the test group log is then
-copied into the overall testsuite log).  This is valid for both
address@hidden and @var{stderr}.
+group log (if the testsuite is run with option @option{-v}, the test
+group log is displayed as the test is run; if the test group later
+fails, the test group log is also copied into the overall testsuite
+log).  This action is valid for both @var{stdout} and @var{stderr}.
+
address@hidden ignore-nolog
+The content of the output is ignored, and nothing is captured in the log
+files.  If @var{commands} are likely to produce binary output (including
+long lines) or large amounts of output, then logging the output can make
+it harder to locate details related to subsequent tests within the
+group, and could potentially corrupt terminal display of a user running
address@hidden -v}.

 @item stdout
 For the @var{stdout} parameter, capture the content of standard output
@@ -22329,6 +22338,14 @@ Writing Testsuites
 parameter, and the standard error capture file will be named
 @file{stderr}.

address@hidden stdout-nolog
address@hidden stderr-nolog
+Like @samp{stdout} or @samp{stderr}, except that the captured output is
+not duplicated into the test group log.  This action is particularly
+useful for an intermediate check that produces large amounts of data,
+which will be followed by another check that filters down to the
+relevant data, as it makes it easier to locate details in the log.
+
 @item expout
 For the @var{stdout} parameter, compare standard output contents with
 the previously created file @file{expout}, and list any differences in
diff --git a/lib/autotest/general.m4 b/lib/autotest/general.m4
index 80d7f66..2e622a5 100644
--- a/lib/autotest/general.m4
+++ b/lib/autotest/general.m4
@@ -2019,8 +2019,11 @@ dnl COMMANDS may contain parameter expansions; expand 
them at runtime.
 # faster than using m4_case, and these are called very frequently.
 m4_define([AT_DIFF_STDERR(stderr)],
          [echo stderr:; tee stderr <"$at_stderr"])
+m4_define([AT_DIFF_STDERR(stderr-nolog)],
+         [echo stderr captured; cp "$at_stderr" stderr])
 m4_define([AT_DIFF_STDERR(ignore)],
          [echo stderr:; cat "$at_stderr"])
+m4_define([AT_DIFF_STDERR(ignore-nolog)])
 m4_define([AT_DIFF_STDERR(experr)],
          [$at_diff experr "$at_stderr" || at_failed=:])
 m4_define([AT_DIFF_STDERR()],
@@ -2028,8 +2031,11 @@ m4_define([AT_DIFF_STDERR()],

 m4_define([AT_DIFF_STDOUT(stdout)],
          [echo stdout:; tee stdout <"$at_stdout"])
+m4_define([AT_DIFF_STDOUT(stdout-nolog)],
+         [echo stdout captured; cp "$at_stdout" stdout])
 m4_define([AT_DIFF_STDOUT(ignore)],
          [echo stdout:; cat "$at_stdout"])
+m4_define([AT_DIFF_STDOUT(ignore-nolog)])
 m4_define([AT_DIFF_STDOUT(expout)],
          [$at_diff expout "$at_stdout" || at_failed=:])
 m4_define([AT_DIFF_STDOUT()],
diff --git a/tests/autotest.at b/tests/autotest.at
index 8992852..ed898cd 100644
--- a/tests/autotest.at
+++ b/tests/autotest.at
@@ -287,6 +287,51 @@ AT_CHECK_AT_TEST([errexit],
    AT_CHECK([grep "1 .* inhibited subsequent" stderr], [], [ignore])],
   [--errexit])

+AT_CHECK_AT([Logging],
+  [[AT_INIT([artificial test suite])
+    dnl intentionally write failing tests, to see what gets logged
+    AT_SETUP([one])
+    AT_CHECK([echo magicstring01], [1], [ignore])
+    AT_CLEANUP
+    AT_SETUP([two])
+    AT_CHECK([echo magicstring02 >&2], [1], [], [ignore])
+    AT_CLEANUP
+    AT_SETUP([three])
+    AT_CHECK([echo magicstring03], [1], [ignore-nolog])
+    AT_CLEANUP
+    AT_SETUP([four])
+    AT_CHECK([echo magicstring04 >&2], [1], [], [ignore-nolog])
+    AT_CLEANUP
+    AT_SETUP([five])
+    AT_CHECK([echo magicstring05], [1], [stdout])
+    AT_CLEANUP
+    AT_SETUP([six])
+    AT_CHECK([echo magicstring06 >&2], [1], [], [stderr])
+    AT_CLEANUP
+    AT_SETUP([seven])
+    AT_CHECK([echo magicstring07], [1], [stdout-nolog])
+    AT_CLEANUP
+    AT_SETUP([eight])
+    AT_CHECK([echo magicstring08 >&2], [1], [], [stderr-nolog])
+    AT_CLEANUP
+    AT_SETUP([nine])
+    echo magicstring09 > expout
+    AT_CHECK([echo magicstring09], [1], [expout])
+    AT_CLEANUP
+    AT_SETUP([ten])
+    echo magicstring10 > experr
+    AT_CHECK([echo magicstring10 >&2], [1], [], [experr])
+    AT_CLEANUP
+]], [], [1], [], [ignore], [],
+  [AT_CHECK([$CONFIG_SHELL ./micro-suite], [1], [ignore-nolog], [ignore-nolog])
+   AT_CHECK([grep '^magicstring' micro-suite.log], [],
+[[magicstring01
+magicstring02
+magicstring05
+magicstring06
+]])])
+
+
 AT_CHECK_AT_TEST([Cleanup],
   [AT_CHECK([test ! -f cleanup.success && test ! -f cleanup.failure])
    AT_CHECK_NOESCAPE([exit $value], [ignore], [$output],
-- 
1.6.1.2