Re: can't always get make to auto create build directory

[David Boyce]

The problem with the $(shell mkdir -p) solution is that it creates the
directory in all cases, notably "make -n" and "make clean". Not exactly
elegant. It's also a wasted process when the directory already exists, it's
mysterious (i.e. you do not see the directory being created) when it didn't
previously exist, and it will not abort the build on failure.

Directory creation within make turns out to be a surprisingly complex
topic, especially in parallel-build scenarios. I've done a good bit of
analysis and tried every technique and the only way I've found to be
completely robust is the order-only prerequisite. Paul's solution is also
parallel-safe if you can accept the problems noted above.

The primary complexity of mkdir within make is when the mkdir -p flag is
used because it causes a race to create the interior (parent) directories
in parallel builds[*]. Below I contribute a recursive make macro, within a
sample makefile, which solves this problem. It's the safest/most portable
method of directory creation I know.

[*] GNU mkdir contains code to address the race condition with a retry but
that's neither elegant nor portable.

-David

% cat Makefile
# The purpose of this macro is to avoid the need for full directory
# paths to be created via mkdir -p, since mkdir -p has an inherent
# race in parallel use cases. Interestingly, though it exists to
# make -p unnecessary, it actually uses -p. Why is that? It's to
# take advantage of a useful side effect of -p which is that it
# doesn't complain if the directory already exists. In other words
# it creates directories one at a time to avoid creating its *own*
# race but still uses mkdir -p to protect itself against races with
# unrelated processes.
# Usage: $(call target_needs_dir,<target>,<dir-path>)
define target_needs_dir
$(eval
$$1: | $$2
$$2: | $$(filter-out .,$$(patsubst %/,%,$$(dir $$2))); mkdir -p $$@
ifneq (,$$(findstring /,$$2))
  $$(call target_needs_dir,$$1,$$(patsubst %/,%,$$(dir $$2)))
endif
)
endef

$(call target_needs_dir,all,foo/bar/baz)
.PHONY: all
all:
ls -ld foo/bar/baz

.PHONY: clean
clean:
$(RM) -r foo

% make
mkdir -p foo
mkdir -p foo/bar
mkdir -p foo/bar/baz
ls -ld foo/bar/baz
drwxrwxr-x 2 dsb users 4096 Jun 29 21:11 foo/bar/baz

On Wed, Jun 29, 2016 at 5:34 PM, LMH <address@hidden> wrote:

> Paul Smith wrote:
>
>> On Wed, 2016-06-29 at 14:03 -0400, LMH wrote:
>>
>>> #create build directory if it doesn't exist
>>> $(BDIR):
>>>          @mkdir -p $(BDIR)
>>>
>>> all: $(BDIR)/SMD2_i386.exe
>>>
>>
>> Here you've created a target $(BDIR), but your "all" target depends
>> only on the object file $(BDIR)/SMD2_i386.exe.
>>
>> Since nothing depends on the actual target $(BDIR), make will never try
>> to build that target.
>>
>> In order for this to work you'd need to define the directory as a
>> prerequisite of the target which needs it, which in this case is
>> $(BDIR)/SMD2_i386.exe, so you'd need to write:
>>
>>    $(BDIR)/SMD2_i386.exe : <other prereqs> $(BDIR)
>>
>> However, you should generally not have targets depend on directories,
>> because make treats them just like any other file when it checks
>> modification times; however directories do not act like normal files
>> when it comes to modification times.
>>
>> I usually just use:
>>
>>    $(shell mkdir -p $(BDIR))
>>
>> but other people prefer order-only prerequisites:
>>
>>    $(BDIR)/SMD2_i386.exe : <other prereqs> | $(BDIR)
>>
>> (note the extra "|" there).  Check the GNU make manual for details.
>>
>> Note these are all specific to GNU make and not portable to other
>> versions of make.
>>
>>
>
> Going back through older versions of some of my makefiles, I find that I
> used to have an "archdir" target defined along with the code to create the
> build directory.
>
>
> archdir: ${BDIR}
> ${BDIR}:
>         @mkdir -p $(BDIR)
>
>
> The archdir target was included in the all target like,
>
>
> all: archdir $(BDIR)/SMD2_i386.exe
>
>
> Somewhere along the line, this archdir target seems to have been left out
> and I guess this is where I started having problems in cases where the
> build directory didn't already exist. The build directory likely did exist
> in many cases, so I probably didn't notice right away or thought it was a
> problem esoteric to the OS I was booted into.
>
> If you don't mind my asking, how does the above "archdir" solution compare
> with what you suggested as far as defining $(BDIR) as a prerequisite for
> $(BDIR)/SMD2_i386.exe? Is there any difference?
>
> Your shell solution is simpler than mine in that it doesn't need to
> actually check if the directory exists and just suppresses the error
> message if it does. I am wondering if there is any value to actually making
> the check, but I can't think of any off hand.
>
> LMH
>
>
>
>
>
>
> _______________________________________________
> Help-make mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/help-make
>