parallel make and the maintainer GNUmakefile

[Ralf Wildenhues]

Hello Jim, all,

I noticed that the GNUmakefile from coreutils/gnulib uses .NOTPARALLEL,
in order to avoid problems with multiple targets listed on the command
line.  It was added here:

| commit 88b47ac69fc7944fb1d4ff008d5741c55a8f279a
| Author: Jim Meyering <address@hidden>
| Date:   Fri Nov 17 10:20:09 2000 +0000
| 
|     (.NOTPARALLEL): Put it here instead, since this file
|     is common to these packages: fileutils, sh-utils, textutils.
...
|     * Makefile.am (.NOTPARALLEL): New target.  Prevent unwanted parallelism.
|     Suggestion from Ulrich Drepper.


The underlying issue is arguably an Automake bug/limitation, which
however seems hard to fix portably in the generated Makefile.
(E.g., a dependency `check: all' would mean that `make check' recurses
the tree twice; for `make -j3 clean all', I don't see any portable way.)

Anyway, while this doesn't matter all that much for recursive makefile
packages like coreutils, unconditional .NOTPARALLEL slows down builds
for nonrecursive ones like git master M4.  How about limiting the impact
to only those cases where it matters?

I have two alternative patches to suggest, one simple-minded one and one
elaborate one: (guess which one I favor? ;-)

1) disable parallel builds if more than one target is listed on the
command line.  This works, in the sense that it still avoids the bug,
but it also practically doesn't provide parallelism even for 'make -j
all', because the automake-generated Makefiles can (and often do)
reinvoke 'make' with some other targets listed on the command line.
For example, in M4, 'make check' causes invocation of
  make  tests/import.la tests/modtest.la tests/shadow.la tests/stdlib.la 
tests/time.la

and these targets could be updated in parallel but won't with this
patch.


2) disable parallel builds if more than one target is listed on the
command line, and at least one of them invokes a recursive target.
This is the gist of the issue, and the patch only avoids this issue
while practically enabling parallelism throughout the build.  It makes
use of a couple of undocumented Automake details, but is pretty safe in
the sense that is trivially backward compatible to older Automake; I
intend to define AM_RECURSIVE_TARGETS in a future Automake so that
relying on undocumented won't be needed any more.

This second patch shaves a quarter off of the time for 'make -j3 all' of
git M4 on a two-way system.  It does expose a parallel ordering bug in
the M4 makefile though (more on this on the m4-patches list).

Of course, if the package author has added targets to Makefile.am which
are recursive, or invoke recursive targets, then they need to be taken
into account now, too: she should add them to ALL_RECURSIVE_TARGETS.

What do you think?  Is this too complicated to be adhered to?

I really like that it even causes things like
  make -j3 clean all check distcheck

to DTRT now!

I've checked that the MAKECMDGOALS, $(word ...) and $(filter ...)
constructs have been supported by GNU make for longer than .NOTPARALLEL
has (each at least for 10 years).

I've also tested the second patch on some of the other GNUmakefile users
like coreutils and Autoconf, without finding problems.

Note that this patch does not address the following set of targets:
  dist-{gzip,bzip2,lzma,xz,tarZ,shar,zip}

That is, out of this list (plus dist and dist-all), it is simply not
possible to use more than one at a time.  This isn't a parallel build
issue, it's a (mis)feature of those rules, and I'd like to eventually
address this separately (but also I don't think that is such a pressing
issue).

Thanks!
Ralf

    GNUmakefile: disable parallelism only for multiple listed targets
    
    * top/GNUmakefile: Only disable parallel builds if multiple
    targets are listed on the command line.

diff --git a/top/GNUmakefile b/top/GNUmakefile
index 17f1e69..2d9b350 100644
--- a/top/GNUmakefile
+++ b/top/GNUmakefile
@@ -107,6 +107,8 @@ abort-due-to-no-makefile:
 endif
 
 # Tell version 3.79 and up of GNU make to not build goals in this
-# directory in parallel.  This is necessary in case someone tries to
-# build multiple targets on one command line.
+# directory in parallel, in case someone tries to build multiple targets
+# on one command line.
+ifneq ($(word 2, $(MAKECMDGOALS)), )
 .NOTPARALLEL:
+endif


    GNUmakefile: disable parallelism only for multiple, recursive targets
    
    * top/GNUmakefile (ALL_RECURSIVE_TARGETS): New macro; allow user
    additions in the Makefile.
    (AM_RECURSIVE_TARGETS): New macro, override only if not provided
    by Automake.
    (.NOTPARALLEL): Only disable parallel builds if multiple targets
    are listed on the command line and at least one of them is
    listed in $(ALL_RECURSIVE_TARGETS).

diff --git a/top/GNUmakefile b/top/GNUmakefile
index 17f1e69..33eb3aa 100644
--- a/top/GNUmakefile
+++ b/top/GNUmakefile
@@ -38,6 +38,9 @@ ifeq ($(_have-Makefile),yes)
 # Make tar archive easier to reproduce.
 export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner
 
+# Allow the user to add to this in the Makefile.
+ALL_RECURSIVE_TARGETS =
+
 include Makefile
 
 # Some projects override e.g., _autoreconf here.
@@ -107,6 +110,18 @@ abort-due-to-no-makefile:
 endif
 
 # Tell version 3.79 and up of GNU make to not build goals in this
-# directory in parallel.  This is necessary in case someone tries to
-# build multiple targets on one command line.
+# directory in parallel, in case someone tries to build multiple
+# targets, and one of them can cause a recursive target to be invoked.
+
+# Only set this if Automake doesn't provide it.
+AM_RECURSIVE_TARGETS ?= $(RECURSIVE_TARGETS:-recursive=) \
+  $(RECURSIVE_CLEAN_TARGETS:-recursive=) \
+  dist distcheck tags ctags
+
+ALL_RECURSIVE_TARGETS += $(AM_RECURSIVE_TARGETS)
+
+ifneq ($(word 2, $(MAKECMDGOALS)), )
+ifneq ($(filter $(ALL_RECURSIVE_TARGETS), $(MAKECMDGOALS)), )
 .NOTPARALLEL:
+endif
+endif