Re: Parallel build dependency locking

[Christopher Sean Morrison]

On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:

> On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:
> > There is already support for processing multiple targets 
> > simultaneously
> > albeit through some form of locking to ensure that each
> > dependency/target is being handled by only one of the -j workers (in
> > that process).  That is the simple nature of a -j parallel build.  All
> > I'm suggesting is a means to extend that same locking mechanism on
> > dependancies being processed by child, sub-process, or otherwise
> > concurrent makes also.
> [snip]
> What you are suggesting is, no doubt, possible but you are asking for
> locking on a target across independent make processes.  Clearly, one
> could implement locking of that form (although doing it on every
> platform that GNU Make supports might be tricky).

That is indeed what I'd be interested in having.  Another alternative 
might be to consider the sub-process makes as not being independent, 
sharing the exec knowledge of their parent via some mechanism 
(environment vars, process id arguments, etc).  It would seem more 
simple to just handle the more general independent process case, but 
considering only sub-process makes and making them dependent or 
otherwise state sharing would even be beneficial.

> However, I think it's the wrong solution to your problem.  It appears
> that what you want to do is speed up your build, and you've chosen to 
> do
> that by running a parallel build.   If you want to do that then there
> are a couple of necessities: you need to make sure that there are no
> missing dependencies and that nothing is built twice.

In the cases I have been involved with, those two concerns are 
generally not a problem to say the least.  Most (including the one in 
question) are 'gnu build system' projects where the dependencies are 
already in place being described via automake and libtool declarations. 
 With those declarations in place, the build proceeds just fine with 
proper dependency ordering and nothing builds twice.  The 
double-compile problem that I was working through was clearly 
undesirable of course and I'd probably even go so far as to say 
building twice is simply 'wrong'.  Mind you, I'm referring to the 
actual compilation of the source to an object file, not reaching the 
Makefile rule and realizing there's nothing to do (e.g. making target 
binary foo depend on binary bar's objects and bar itself -- when 
processing foo's dependencies, bar's objects are listed explicitly and 
indirectly through bar's rule).

The 'problem' is perhaps just a limitation of the make logic being 
exported by automake and presumptions on possible target dependancies 
(e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are always traversed 
in order one at a time).  Since it processes each program/library in 
order, if there are a lot of 'em (hundreds as there are in my case), 
parallel build performance is crippled.  It ends up compiling an 
object, linking a binary, compiling an object, linking a binary over 
and over effectively serializing the build process.  All of the objects 
are actually independent (and specified as such via the automake rules) 
so they can go in parallel, as are all the binaries so they could all 
link in parallel so long as the corresponding objects are done.  
Minimally, I now know that it 'can' be done, since I just worked in my 
own make traversal logic that does just that -- it now compiles all 
object files in parallel and then links all binaries together in 
parallel.

That said, I was only able to do that by avoiding that separate make 
process that automake or autoconf injected on the 'all' target.  While 
I do heavily concede that it's probably more a design limitation of the 
automake/autoconf-generated Makefiles since they could just as easily 
provide a similar 'fast' make path like I wrote, I believe that the 
cross-process dependency locking would allow much greater potential for 
more effective parallel builds without requiring too much hoop-jumping 
and customization on the Makefile writer.  It would also produce the 
exact same end result with no detriment to other parallel or 
non-parallel builds (that come to mind).

It should indeed only 'build things once'.  I'm suggesting extending 
the same logic that exists with the -j option across make instances.  
So if make -j2 works, so would running make& twice and it should be 
possible to make it execute with practically identical behavior.

Sean