Re: make behavior WRT background processes

[Mervyn Keene]

It looks like my original message got some kind of rich-text disease so
here's a retry in plain-text format (I hope):

I'm looking for insight in the area of how make handles background
processes started from recipes.  Let me start with a summary of my
real-word scenario: we have a Linux makefile which is large, complex and
proprietary but in the end all it does is compile a bunch of objects and
link them together. After that final binary is linked there are quite a few
post-actions performed on it involving objdump, objcopy, and other
ELF-wrangling tools. Because there are so many of these post-tasks I've
wrapped them up in a Python script. So the basic workflow, from make's POV,
looks like:

- compile many objects
- link them together into blah.exe
- run a python script which operates on blah.exe

One of the sub-tasks invoked from the Python script is (a) quite slow and
(b) rarely important (it's a disassembly which is occasionally useful in
debugging but is never used within the build per se). I'd like to add a
"fast build" mode in which the disassembly is spun off in the background,
the build returns without waiting for it, and that file shows up later (or
not, if it fails). Of course there would be a production mode which builds
it synchronously too.

I've implemented this but make appears to wait for the background process
anyway. In other words the disassembler definitely goes into the
background, subsequent tasks run immediately, everything gets finished ...
 but then make waits for the background process so all the benefit is lost.
I've put together a simple test case (below) but the problem isn't
reproduced by it and w ithout being able to reproduce I'm a little stuck.
AFAICT this test case faithfully reproduces the situation from a process
POV but it behaves "correctly" by not waiting:


% cat Makefile
.PHONY: all
all: fast1 slow1 fast2
@echo "MAKE DONE"

fast1:
@echo "Done with $@"

fast2:
@sleep 1
@echo "Done with $@"

slow1:
python -c "import os; os.system('(sleep 5; echo PYTHON JOB DONE) &')"
@echo "Done with $@"

% make -j
python -c "import os; os.system('(sleep 5; echo PYTHON JOB DONE) &')"
Done with fast1
Done with slow1
Done with fast2
MAKE DONE

[time passes ...]


% PYTHON JOB DONE

Note that make says "MAKE DONE" and exits back to the shell, then 5 seconds
later the "PYTHON JOB DONE" arrives which is exactly the behavior desired.
But in the real makefile the "PYTHON JOB DONE" shows up before "MAKE DONE"
meaning that make has waited for it.  Does anyone have a theory about how
this could happen? I've tried the double-fork technique but it doesn't seem
to help.



On Mon, Feb 19, 2018 at 8:36 AM, Mervyn Keene <address@hidden>
wrote:

> I'm looking for insight in the area of how make handles background
> processes started from recipes.
> ​Let me
>  start
> ​ with​
> a
> ​summary
>  of
> ​my
>  real-word scenario: we have a Linux makefile which is large, complex and
> proprietary but in the end all it does is compile a bunch of objects and
> link them together. After
> ​that
>  final binary is linked there are quite a few post-actions performed on it
> involving objdump, objcopy, and other ELF-wrangling tools. Because there
> are so many of these post-tasks I've wrapped them up in a Python script. So
> the basic workflow
> ​, from make's POV,​
> looks like:
>
> ​- ​
> compile many objects
> ​- ​
> link them together
> ​ into blah.exe​
>
> ​- ​
> run a python script
> ​ which operates on blah.exe​
>
>
> One of the sub-tasks invoked from the Python script is (a)
> ​quite
>  slow and (b)
> ​rarely
>  important (it's a disassembly which is occasionally useful in debugging
> but is never used within the build per se). I'd like to add a "fast build"
> mode in which the disassembly is spun off in the background, the build
> returns without waiting for it, and
> ​that file
>  shows
> ​up
>  later (or not
> ​, if it fails​
> ). Of course there would be a production mode
> ​which builds it
>  synchronously too.
>
> I've implemented this but make
> ​still ​
> appears to wait for the background process anyway. In other words
> ​the disassembler
>  definitely goes into the background, subsequent tasks run, everything
> ​is finished
>  ...
> ​but
>  then make waits for the background process
> ​ so all the benefit is lost​
> . I've put together a simple test case
> ​(below) ​
> but the problem isn't reproduced
> ​ ​
> by it
> ​ and w
> ithout being able to reproduce I'm a little stuck. AFAICT
> ​this
> test case faithfully reproduces the situation from a process POV but it
> behaves "correctly"
> ​ by not waiting:
>
>
> % cat Makefile
> .PHONY: all
> all: fast1 slow1 fast2
> @echo "MAKE DONE"
>
> fast1:
> @echo "Done with $@"
>
> fast2:
> @sleep 1
> @echo "Done with $@"
>
> slow1:
> python -c "import os; os.system('(sleep 5; echo PYTHON JOB DONE) &')"
> @echo "Done with $@"
>
> % make -j
> python -c "import os; os.system('(sleep 5; echo PYTHON JOB DONE) &')"
> Done with fast1
> Done with slow1
> Done with fast2
> MAKE DONE
>
> [time passes ...]
>
>
> % PYTHON JOB DONE
>
> Note that make says "MAKE DONE"
> ​and exits back to the shell,
> then
>  ​
> 5 seconds later
> ​
>  the "PYTHON JOB DONE"
> ​ arrives​
> . This is exactly
> ​the behavior
>  desired but in the real makefile the "PYTHON JOB DONE" shows up before
> "MAKE DONE"
> ​ meaning that make has waited for it.
> Does anyone have a
> ​theory
>  about
> ​how this could happen
> ​?​
>
>
>
>