demonstrating trivial issue with make running in parallel

[rpjday]

(can't reply to thread from yesterday as i'm doing this from webmail and
have no access to what's already been downloaded to my home system.)

asked about weird parallelization issues in make structure in the code
base here:

https://github.com/Juniper/JP4Agent

and I wanted to demo a really trivial example to show when this issue
would pop up -- if people want to improve this example, that would
be ducky; I will probably write it up and post it on my wiki as a lesson
to others.

mk1:
====

  first, a simple example of targets that can be processed in parallel:


COUNTERS = count1 count2

.PHONY: top $(COUNTERS)

top: $(COUNTERS)

count1:
        for i in $$(seq 5) ; do \
                echo "$@ $$i" ; \
                sleep 1 ; \
        done

count2:
        for i in $$(seq 5) ; do \
                echo "$@ $$i" ; \
                sleep 1 ; \
        done


now in the above, there is no dependency relation between count1 and count2,
so they *can* run in parallel. if one does not ask for that, they won't be:

$ make -f mk1
for i in $(seq 5) ; do \
        echo "count1 $i" ; \
        sleep 1 ; \
done
count1 1
count1 2
count1 3
count1 4
count1 5
for i in $(seq 5) ; do \
        echo "count2 $i" ; \
        sleep 1 ; \
done
count2 1
count2 2
count2 3
count2 4
count2 5
$


but if one *does* ask for parallelism, one gets it:

t$ make -f mk1 -j 2
for i in $(seq 5) ; do \
        echo "count1 $i" ; \
        sleep 1 ; \
done
for i in $(seq 5) ; do \
        echo "count2 $i" ; \
        sleep 1 ; \
done
count1 1
count2 1
count1 2
count2 2
count1 3
count2 3
count1 4
count2 4
count1 5
count2 5
$

so far, so good, yes? and this is because there is no relationship
between targets count1 and count2 in this case. moving on …

mk2:
====


COUNTERS = count1 count2

.PHONY: top $(COUNTERS) count

top: $(COUNTERS)

count1 count2: count

count:
        for i in $$(seq 5) ; do \
                echo "$$i" ; \
                sleep 1 ; \
        done


this situation is clearly different in that count1 and count2 are
now "related" via their mutual dependency on "count" so that (AIUI)
that target should be processed only once, no matter what:


$ make -f mk2
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
1
2
3
4
5
$

t$ make -f mk2 -j 2        <-- even with parallelism, only once
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
1
2
3
4
5
$

so this seems fine, but that's not what appears to be happening
in that JP4agent Makefile. rather than setting up proper dependencies,
the later rules simply call make manually for subdirectories, as in:


INSTALL_COMPONENTS = $(COMPONENTS:%=install-%)
.PHONY: install $(INSTALL_COMPONENTS)
install: $(INSTALL_COMPONENTS)
$(INSTALL_COMPONENTS):
        echo $(DESTDIR) $(prefix) $(sysconfdir)
        make install -C $(@:install-%=%)

which means any protection for parallelism is totally lost, correct?
I believe I can demonstrate that with …

mk3:
====

COUNTERS = count1 count2

.PHONY: top $(COUNTERS) count

top: $(COUNTERS)

count1 count2:
        $(MAKE) -f mk3 count     <--- the important bit

count:
        for i in $$(seq 5) ; do \
                echo "$$i" ; \
                sleep 1 ; \
        done


so, in the above, rather than defining proper dependencies, by
manually calling the make command in a rule (is that the right way
to do that?), first, if i invoke this makefile normally, it runs
targets serially:


$ make -f mk3
make -f mk3 count
make[1]: Entering directory '/home/rday/t'
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
1
2
3
4
5
make[1]: Leaving directory '/home/rday/t'
make -f mk3 count
make[1]: Entering directory '/home/rday/t'
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
1
2
3
4
5
make[1]: Leaving directory '/home/rday/t'



but if ask for parallelism, then bad things happen:


$ make -f mk3 -j 2
make -f mk3 count
make -f mk3 count
make[1]: Entering directory '/home/rday/t'
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
make[1]: Entering directory '/home/rday/t'
for i in $(seq 5) ; do \
        echo "$i" ; \
        sleep 1 ; \
done
1
1
2
2
3
3
4
4
5
5
make[1]: Leaving directory '/home/rday/t'
make[1]: Leaving directory '/home/rday/t'


i submit that this is what is happening based on that top-level
Makefile … any thoughts on whether my explanation above would seem
to match what is happening there? is there a better/cleaner way to
describe this?

rday