Re: Parallel build dependency locking

[Christopher Sean Morrison]

> > Sounds like
> > it's fairly safe to presume that there's no concurrent make file/job
> > locking across to submakes (e.g. some file-based semaphore locking,
> > shared memory locking, etc).  Is this a bug/feature request worth
> > reporting?
>
> No. ;-)
>
> Seriously, if you are building the same target more than once then
> it's a bug in your makefile and you normally wouldn't want make to
> hide it from you.

That may very well be true, but in this particular instance, the 
workaround/solution to use the all-am did nothing to change which 
targets were reached.  It merely avoided a secondary invocation of make 
(that did the exact same thing, reached the same targets/rules in 
parallel, etc).

There is already support for processing multiple targets simultaneously 
albeit through some form of locking to ensure that each 
dependency/target is being handled by only one of the -j workers (in 
that process).  That is the simple nature of a -j parallel build.  All 
I'm suggesting is a means to extend that same locking mechanism on 
dependancies being processed by child, sub-process, or otherwise 
concurrent makes also.

Basically, that would be a means to share the current processing 
environment across concurrent makes.  It should only impact the -j 
parallel builds if done correctly, allowing the workers that have 
completed with one dependency to progress to the next set of rules 
while remaining coherent with respect to each other's progress.  This 
would be a significant boost on my larger SMP systems where a -j32 or 
-j200 or larger is not unheard of -- with proper locking, you could 
even start to think about structuring together makefiles that perform 
simultaneous multi-directory builds (e.g. build all your libraries in 
parallel before building your binaries (also in parallel)) or more 
complicated rule resolutions.  Heck, one could even invoke make 
directly N times (without -j and with proper external locking) and have 
it successfully build in parallel instead of having the various 
processes stomp on each other's results (quite useless).

It's a feature that I've heard from others as well that work on larger 
SMP systems with make-based projects.  In general, it's just requests 
for better support for parallel builds in general and the 
aforementioned suggestion would be an overall improvement.  For now, 
I've hacked around it enough so that almost our entire package will now 
compile and link in parallel, significantly reducing our compile times 
-- so I'm quite happy.  ;-)  The only thing that I didn't attempt was 
processing multiple independent directories simultaneously.. I'll save 
that for another rainy day.

Cheers!
Sean