Re: Using GNU make jobserver from GCC

[Jan Hubicka]

> On Mon, 2014-05-19 at 06:39 +0200, Jan Hubicka wrote:
> > > On Mon, 2014-05-19 at 00:48 +0200, Jan Hubicka wrote:
> > > > I wonder if thre can be any additional support from GNU's make to make
> > > > "+" unnecesary and if it would be possible to spearate client side of
> > > > GNU Make's jobserver into separate file that can be spossibly shared
> > > > in between GNU make and GCC - perhaps via liberty library?
> > > 
> > > For the second question, the short answer is "no, not right now".
> > > Unfortunately the jobserver feature is sort of embedded into make and
> > > not easily extracted.  That doesn't mean it couldn't be extracted, but
> > > as it's written today it's not entirely clear what the higher-level
> > > interface would need to be.  Some investigation would need to be
> > > undertaken.  However, the POSIX-based interface for jobserver support is
> > > pretty trivial and I'd be surprised if it's worthwhile to extract it: it
> > > just reads a byte and writes a byte.  If you want to try to allow for
> > > jobserver on Windows as well, which uses a very different method of
> > > tracking jobs than the POSIX ports, that would be another story.
> > 
> > Since GCC supports Windows, too, I would love to have generic solution.  To 
> > be
> > honest, GCC currently uses fork for the parallelism that probalby won't make
> > Windows fly, but threaded implementation is possible, too (just extra work 
> > to
> > make streaming thread-safe).
> 
> Threading vs. non-threading is not relevant.  Make doesn't use threads,
> either.
> 
> > Ideally the first loop calling wait_for_child can be replaced by libiberty
> > function that will just sleep until next job is available.
> 
> All you have to do is a blocking read(2) of a single byte on the
> jobserver read file descriptor.  If one is available, you'll return.  If
> not, you'll block until one becomes available.
> 
> The tricky part is that while you're blocked, you won't be reaping
> children so you won't be noticing that they have died, and you won't be
> giving tokens back to the jobserver.  This issue is one of the major
> problems with POSIX asynchronous notification models.  You can read more
> about how the jobserver works, and works around this issue, here:
> 
> http://make.mad-scientist.net/jobserver.html
> 
> Of course, another option is to have the waitpid() done in another
> thread; that would also solve the problem.

Yeah, I read the article. I assume real world implementation will be slightly 
trickier,
since one needs to take care i.e. of cases where GCC crashes or return early, 
right?
> 
> > > For the first question, the reason "+" is needed is that some programs
> > > that make invokes do not behave well if there are extra file descriptors
> > > open (other than stdin, stdout, stderr) when they are invoked.  As a
> > > result, make is careful to close all open file descriptors other than 0,
> > > 1, and 2 before invoking any program that does not need them.  Recursive
> > > makes need them, obviously, to support the jobserver, so any command
> > > that contains $(MAKE) or uses the "+" token keeps the file descriptors.
> > > Any other program has them closed before being invoked.
> > 
> > Hmm, I wonder if we can't pass necessary info in environment var to 
> > re-connect
> > to jobserver?
> 
> It's not possible to "re-open" a pipe in a sub-process once it's been
> closed by the parent.  There is no info that could be passed in the
> environment that could allow this to happen.
> 
> The only way this could be done is if we switched from regular pipes to
> named pipes, but that's much less portable (and has its own issues, such
> as where does the named pipe go and how is it cleaned up, etc.)
> 
> > I suppose adding gcc to the list of commands that keeps
> > descriptors is not the coolest idea...
> 
> The rule today is completely trivial: make looks at the _unexpanded_
> command line for the string $(MAKE) or ${MAKE}.  If found it's
> considered a recursive make invocation.  If not, not.  This is pretty
> safe because MAKE is a reserved variable name.
> 
> This won't work with "gcc".  First, make would have to search the
> command line AFTER expansion is complete, not before.  Second, there's
> no guarantee that the program will be called "gcc".  It might be called
> "g++".  Or "x86_64-linux-gnu-gcc".  Or something else entirely.  And
> finally, there's no guarantee that a command that contains the string
> "gcc" anywhere is actually invoking GCC.
> 
> Plus, it's possible other commands besides GCC might want to participate
> in the jobserver as well.

Yeah, do we have some options except for named pipes on hosts where named pipes
are supported?

Honza