Re: looking for assistance with "parallel" makefile, willing to pay

[Kaz Kylheku (gmake)]

On 2020-02-25 15:45, Robert P. J. Day wrote:
>   if someone wants to isolate the problem and fix it, i'd be a happy
> camper as i have lots of other stuff to work on.
>
>   thoughts?

If multiple targets in the top-level makefile are recursively invoking
make in src/pi/protos, running the install target, that's a problem.

The copies will interfere with each other, because cp is doing something
like

    unlink('target');
    fd = open('target', O_WRONLY | O_CREAT | O_EXCL);
    write(fd, data)

if another copy is running in parallel and creates the file between
the above unlink and open, then that will fail with EEXIST.

One possible way to fix it is to use a lock directory as a mutex:

At the start of the operation do something like:

   .PHONY: install
   install:
       mkdir -p $(DESTDIR) # ensure we have a place where .lockdir goes
       while ! mkdir $(DESTDIR)/.lockdir ; do sleep 1 ; done

       # do all the file file copying

       rmdir .lockdir

mkdir fails when a directory exists, and I think there
is enough synchronization in NFS over directory creation that
it even works there (whereas lock files do not necessarily).

There is a risk of stale .lockdir jamming up the build, though.

It could be pre-emptively removed by some top-level prepare step
(that is not itself parallelized).

That rule could use a general stamp file so that it doesn't
wastefully repeat all the file copying. Serializing it is one
thing, but it's still wasteful to repeatedly do it.

P.S. here's a more robust lock directory implementation, in Lisp. :)
That has a timeout, and checks for the situation that making
the directory fails for an error other other than 17/EEXIST

http://www.kylheku.com/cgit/tamarind/tree/lockdir.tl