Re: Parallel build dependency locking

[Christopher Sean Morrison]

Boris,

Thanks for the reply!  One of your comments pushed me into the right 
direction; I believe I have it working now and maybe even understand 
what was happening that caused problems.  I simplified the fast target 
rule to just simply build without recursing since the recursion was not 
strictly relevant to that particular library example (since I was 
already fully recursed with no remaining SUBDIRS).  That said, it does 
seem like a possible contention bug or perhaps just undesirable 
behavior.  I simplified the 'fast' target rule to just:

fast: $(FAST_OBJECTS) all

and the problem still remained.  Yet when I removed the 'all' 
dependency, the problem went away.  Looking at the automake-generated 
'all' dependency, the problem becomes more clear:

all:
        $(MAKE) $(AM_MAKEFLAGS) all-am

So there is the second invocation of make coming into play that hits up 
the 'all-am' rule, which eventually includes the same objects included 
in FAST_OBJECTS.  So when the first process is working on the last 
couple FAST_OBJECTS objects, the second one continues to the 'all' 
rule, causes make to reinvoke, and ends up picking up where the build 
seemingly left off at which ultimately clobbers the first.  Sounds like 
it's fairly safe to presume that there's no concurrent make file/job 
locking across to submakes (e.g. some file-based semaphore locking, 
shared memory locking, etc).  Is this a bug/feature request worth 
reporting?

The fix/workaround for this particular example was rather simple.  The 
second invocation can be avoided by using 'all-am' as the 'fast' rule 
dependency instead of 'all'.  Once I did that, the sub-make is skipped. 
 The entire recursion and fast build subsequently works great.  For our 
particular software suite which includes over 400 binaries and a couple 
dozen libraries, this saves quite a bit of time (wish the automake 
folks provided such a facility automatically).  Thanks again.

Cheers,
Sean


On Jun 8, 2005, at 12:15 PM, Boris Kolpackov wrote:

> First off, there are no threads; GNU make issues (certain) commands in
> parallel when run with -j flag.
>
> Christopher Sean Morrison <address@hidden> writes:
>
> > fast: fast-local $(FAST_OBJECTS) fast-recursive all
> >
> > fast-recursive:
> >         @list='$(SUBDIRS)'; for subdir in $$list; do \
> > 		test "$$subdir" = . || (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) 
> > fast);
> > \
> >         done
>
> Perhaps you are building the same object file from two concurrent make
> invocations. One is current and the other is the result of the
> fast-recursive rule?
>
> > Compared to wavelet.c, vers.c is very trivial so I can see how a 
> > thread
> > working on vers.lo would finish before the wavelet.lo target's thread
> > gets a chance to even start writing.
>
> > That would only make sense,
> > though, if the OBJECTS are not being properly locked by make on a
> > parallel build
>
> GNU make doesn't do any locking, AFAICT.
>
> > -- and it doesn't seem to explain why the libbn.la
> > target with the same target dependancies doesn't exhibit the same
> > problem.
>
> Perhaps it's because fast-recursive rule is not built?
>
>
> hth,
> -boris