Re: make internals and parallelization

[Michael Muratet]

On Aug 12, 2009, at 11:13 AM, Paul Smith wrote:

> On Wed, 2009-08-12 at 10:23 -0500, Michael Muratet wrote:
> > I have been searching the list archives looking for information on  
> > how
> > make implements parallelization with make -j. From a most global  
> > view,
> > it appears that there is a data structure of dependancies and  
> > multiple
> > branches from the root are candidates for parallel processing. I
> > haven't seen any mention of a developer's manual, is there one? Can
> > anyone point me to some documentation or explain the process?

Paul

> There is no developers manual.  I'm not aware of too many open source
> projects that provide such a thing: there's the source code and the
> comments therein after all :-).
Me either, but it never hurts to ask. I will check out the latest  
version and start reading.
> It's not clear what exactly you're trying to understand so it's hard  
> to
> explain the process.
I'm looking at a very complicated make file that runs data analysis  
tools as opposed to building code and I'm trying to understand if it's  
possible for racing to occur, i.e., a dependancy that's obscured and  
so one branch of the process crashes because it needed something in  
another branch that hasn't completed.
> There's a discussion of GNU make's jobserver algorithm on my page (see
> the signature), but that talks only about how make ensures that no  
> more
> than N jobs are run for "make -jN".
I read the discussion and found it very interesting. The answers to my  
questions may lie within, let me digest it for awhile.
> Make doesn't implement any sort of threading.  Since make spends  
> most of
> its time just waiting for child processes to complete, when you're
> running in parallel it simply uses that "down time" to determine the
> next job that could be run.  When it finds one, it will start that job
> and begin looking through its data structures for the next one, etc.
If and how it threads were among my questions. So when it starts the  
nth job, how does it do that? We've had some problems with vfork in  
the past and I had some posts about it a few weeks ago. I'm 99% sure  
it's a memory problem and I'm chasing that, but is make using vfork to  
start jobs? How does it start processes.

Thanks for the help

Mike

> --
> -------------------------------------------------------------------------------
> Paul D. Smith <address@hidden>          Find some GNU make tips at:
> http://www.gnu.org                      http://make.mad-scientist.net
> "Please remain calm...I may be mad, but I am a professional." --Mad  
> Scientist