bug#9245: FreeBSD make in concurrent mode report spurious success in automake-generated tests harness

[Stefano Lattarini]

On Friday 05 August 2011, Stefano Lattarini wrote:
> Here is a brief illustration of the issue:
> 
>   $ cat > Makefile.am <<'END'
>   AUTOMAKE_OPTIONS = foreign parallel-tests -Wall -Werror
>   TEST_LOG_COMPILER = false
>   TESTS = foo.test bar.test
>   END
>   $ touch foo.test bar.test
>   $ autoreconf -vi && ./configure
>   ...
> 
>   $ freebsd-make -j2 check; echo %% STATUS = $? %%
>   freebsd-make  check-TESTS
>   FAIL: foo.test
>   FAIL: bar.test
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   %% STATUS = 0 %%
> 
> It works as expected with GNU make and NetBSD make though:
> 
>  $ gmake -j2 check; echo %% STATUS = $? %%
>   gmake  check-TESTS
>   gmake[1]: Entering directory `/tmp/...'
>   gmake[2]: Entering directory `/tmp/...'
>   FAIL: foo.test
>   FAIL: bar.test
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   gmake[2]: *** [test-suite.log] Error 1
>   gmake[2]: Leaving directory `/tmp/...'
>   gmake[1]: *** [check-TESTS] Error 2
>   gmake[1]: Leaving directory `/tmp/...'
>   gmake: *** [check-am] Error 2
>   %% STATUS = 2 %%
> 
> 
>   $ netbsd-make -j2 check; echo %% STATUS = $? %%
>   --- check-am ---
>   netbsd-make  check-TESTS
>   --- check-TESTS ---
>   --- foo.log ---
>   --- bar.log ---
>   FAIL: bar.test
>   --- foo.log ---
>   FAIL: foo.test
>   --- test-suite.log ---
>   ====================
>   2 of 2 tests failed
>   See ./test-suite.log
>   ====================
>   *** [test-suite.log] Error code 1
>   1 error
>   netbsd-make: stopped in /tmp/...
>   *** [check-TESTS] Error code 2
>   1 error
>   netbsd-make: stopped in /tmp/...
>   *** [check-am] Error code 2
>   1 error
>   netbsd-make: stopped in /tmp/...
>   %% STATUS = 2 %%
> 
> The problem is present both in automake 1.11.1 and in the developement
> version from the 'test-protocols' branch.
> 
OK, the attached hacky patch seems to fix the bug.  I have no idea why
exactly it is so yet, so we might want to wait to apply the patch until
we have fully understood the reasons of the original failure, and why
the proposed patch seems to fix it.

Regards,
  Stefano

From e7568ee0cfaccc5c9998275a5d93643a49ea533d Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Fri, 12 Aug 2011 18:29:28 +0200
Subject: [PATCH] parallel-tests: no more spurious successes for FreeBSD make

Work around a bug of FreeBSD make bug that was causing the
automake-generated "check" target to complete with success
even if some tests failed; this happened only when FreeBSD
make was run in concurrent mode (as in, e.g., "make -j2
check").  The bug is not present in NetBSD make.

This change fixes automake bug#9245.

* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
more "safe" (and apparently redundant) idiom to exit with error,
so that the non-zero exit status is picked up also by FreeBSD
make when it's running in concurrent mode.
* NEWS: Update.
* tests/parallel-tests-concurrency-bug9245.test: New test.
* tests/Makefile.am (TESTS): Update.
---
 ChangeLog                                     |   17 ++++++++
 NEWS                                          |    4 +-
 lib/Automake/tests/Makefile.in                |    2 +-
 lib/am/check.am                               |    4 +-
 tests/Makefile.am                             |    1 +
 tests/Makefile.in                             |    3 +-
 tests/parallel-tests-concurrency-bug9245.test |   55 +++++++++++++++++++++++++
 7 files changed, 82 insertions(+), 4 deletions(-)
 create mode 100755 tests/parallel-tests-concurrency-bug9245.test

diff --git a/ChangeLog b/ChangeLog
index af2556f..f7a3565 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,20 @@
+2011-08-12  Stefano Lattarini  <address@hidden>
+
+       parallel-tests: no more spurious successes for FreeBSD make
+       Work around a bug of FreeBSD make bug that was causing the
+       automake-generated "check" target to complete with success
+       even if some tests failed; this happened only when FreeBSD
+       make was run in concurrent mode (as in, e.g., "make -j2
+       check").  The bug is not present in NetBSD make.
+       This change fixes automake bug#9245.
+       * lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
+       more "safe" (and apparently redundant) idiom to exit with error,
+       so that the non-zero exit status is picked up also by FreeBSD
+       make when it's running in concurrent mode.
+       * NEWS: Update.
+       * tests/parallel-tests-concurrency-bug9245.test: New test.
+       * tests/Makefile.am (TESTS): Update.
+
 2011-08-08  Stefano Lattarini  <address@hidden>
 
        test defs: more environment cleanup
diff --git a/NEWS b/NEWS
index 29c552b..5086f75 100644
--- a/NEWS
+++ b/NEWS
@@ -23,7 +23,9 @@ Bugs fixed in 1.11.0a:
 * Bugs introduced by 1.11:
 
   - The `parallel-tests' test driver works around a GNU make 3.80 bug with
-    trailing white space in the test list (`TESTS = foo $(EMPTY)').
+    trailing white space in the test list (`TESTS = foo $(EMPTY)'), and
+    does not report spurious successes when used with concurrent FreeBSD
+    make (e.g., "make check -j3").
 
   - The `silent-rules' option now also silences all compile rules if dependency
     tracking is disabled.  Also, when `silent-rules' is not used, the output 
from
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 2f553ed..1cb438c 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -405,7 +405,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          col="$$red";                                                  \
        fi;                                                             \
        echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
-       $$exit
+       $$exit || (exit 1); exit 1
 
 # Run all the tests.
 check-TESTS:
diff --git a/lib/am/check.am b/lib/am/check.am
index 0b54312..75f8bba 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -224,7 +224,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          col="$$red";                                                  \
        fi;                                                             \
        echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
-       $$exit
+## The apparently redundant and repeated "exit 1" works around a bug in
+## FreeBSD make (present only when running in concurrent mode).
+       $$exit || (exit 1); exit 1
 
 RECHECK_LOGS = $(TEST_LOGS)
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b545dda..a1f7818 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -605,6 +605,7 @@ parallel-tests-log-override-1.test \
 parallel-tests-log-override-2.test \
 parallel-tests-log-override-recheck.test \
 parallel-tests-log-compiler-example.test \
+parallel-tests-concurrency-bug9245.test \
 parse.test \
 percent.test \
 percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index fe44a40..7bb6586 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -882,6 +882,7 @@ parallel-tests-log-override-1.test \
 parallel-tests-log-override-2.test \
 parallel-tests-log-override-recheck.test \
 parallel-tests-log-compiler-example.test \
+parallel-tests-concurrency-bug9245.test \
 parse.test \
 percent.test \
 percent2.test \
@@ -1274,7 +1275,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          col="$$red";                                                  \
        fi;                                                             \
        echo "$$msg" | $(am__text_box) "col=$$col" "std=$$std";         \
-       $$exit
+       $$exit || (exit 1); exit 1
 
 # Run all the tests.
 check-TESTS:
diff --git a/tests/parallel-tests-concurrency-bug9245.test 
b/tests/parallel-tests-concurrency-bug9245.test
new file mode 100755
index 0000000..78bd98d
--- /dev/null
+++ b/tests/parallel-tests-concurrency-bug9245.test
@@ -0,0 +1,55 @@
+#! /bin/sh
+# Copyright (C) 2009, 2010, 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Look for a bug where FreeBSD make in concurrent mode reported success
+# even when the Automake-generated parallel testsuite harness failed.
+# See automake bug#9245.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_COMPILER = false
+TESTS = foo.test bar.test
+END
+
+: > foo.test
+: > bar.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+# Some make implementations don't grok the `-j' option.
+$MAKE -j1 || Exit 77
+
+for j in '' -j1 -j2; do
+  $MAKE $j check && Exit 1
+  $MAKE $j recheck && Exit 1
+  TESTS=foo.test $MAKE $j -e check && Exit 1
+  TEST_LOGS=foo.log $MAKE $j -e check && Exit 1
+  rm -f test-suite.log
+  $MAKE $j test-suite.log && Exit 1
+  test -f test-suite.log
+done
+
+:
-- 
1.7.2.3