Re: The smart way to use make in a directory tree

[David Boyce]

On Thu, Jul 15, 2010 at 11:07 PM, Josh Bialkowski <address@hidden> wrote:
> Now my question: I'm looking for a "smart" way to handle the make structure
> for the different projects, so that when I "make" a project (i.e. projA) it
> will remake any of the libraries it depends on (i.e. libs A, B, D) if any of
> their dependencies have changed. Right now I have one makefile in each
> subdirectory (i.e. libA/makefile libB/makefile projA/makefile). In order to
> build project A I have to manually cd into each of the libraries and make
> them (if I've changed any code in there) and then cd into the project
> directory and rebuild that project.

This could get long but I'm in the mood to explain.

You're lucky because you haven't yet had the chance to form bad
habits. The obvious solution, chosen by most people at least
historically, is to create a "master Makefile" at the root of the tree
which iteratively cd's into each subdir and runs make there, but this
would be the bad habit of which I speak. The first thing you should do
is google for Recursive Make Considered Harmful and read it, even
though it's a little dated. In fact, make is the one place I know of
where recursion is not the elegant, preferred solution.

The next thing is to step back and consider the 'impedance mismatch'
between the way you think of your project and the way make does. You
may see it as a tree structure, and you may say things like "I want
make to build the libraries first" or "I want it to visit directories
in such and such order". But to make there's just a set of files
(note: set, not list - there is no structure or order to the way make
encounters files) and a web of dependency information between them.
The upshot is that if you want libfoo.a to be updated before progX
links with it, then the file 'progX' needs to show libfoo.a as a
prerequisite. But the bigger picture point is that though the tree
structure is a convenient organizing principle for the human mind,
nothing good comes of forcing make to look at the project this way.
Make does not work directory-by-directory, it works target-by-target.

It follows from this that the same instance of make needs to be aware
of both libfoo.a and progX, and thus that the only way to build a
project reliably is to use a single instance of make which knows about
the whole thing. This is the basic point that the RMCH paper is
making, though in somewhat more detail.

Here's where you get the opportunity to say "Hey, I'm a scientist, not
a software engineer. My cure for cancer can't wait while I get the
build model right!" Which would be a perfectly legitimate choice and
sometimes the right one - just be aware that the build will continue
to be a weakness and once in a while you'll have a bug which turns out
to be because you forgot to rebuild the library with the fixed code
first.

Assuming you want to get it right, read on. We come next to another
mismatch, between the way make works and the way it should work. By
default it looks for makefiles in the current directory, because the
original author of make was himself a victim of the
one-directory-at-a-time fallacy. What it should do (IMHO) is search
through parent directories as well, but it doesn't. Thus, you'll need
a file called "Makefile" in each directory where you might type
"make", while at the same time maintaining just one logical makefile.

There are 3 ways to solve this: (1) if you're sure you'll always be on
a system which supports symlinks (and even Windows is said to do so in
recent versions), you can keep everything in a master Makefile at the
project root with a symlink to it from each subdir, or (2) you can
have each sub-Makefile be a one-liner which includes the master. These
methods are simple and robust; the only downside is that you don't get
to maintain the fiction that the description of how to build "lib2" is
in .../lib2/Makefile, because it's all in a single monolithic file.
This closely models the truth but may not be what people are used to.
Method (3) would be to keep an actual set of Makefiles, each
containing the description of how to build the stuff in its directory,
and have them include each other in a cyclical arrangement. Typically
the master Makefile simply includes all sub-makefiles and they in turn
include the master, with the same kinds of conditional protections
used in header files to prevent infinite loops. All these methods
result in a single logical Makefile and the same behavior; the
difference is mostly a question of style.

Last, what goes inside these Makefiles? Standard makefile syntax but
now, since everything is considered together, you'll need to be
careful about namespace in two ways:

1. "Local" variables need to be uniqified, e.g. $(lib1_objects)
instead of $(objects).

2. *Every path mentioned must be absolute*. It's a good idea to prefix
paths with a ${PROJECT_ROOT} variable which is determined at the top
of the master makefile via something like this:

ifndef PROJECT_ROOT
export PROJECT_ROOT := $(dir $(abspath $(lastword ${MAKEFILE_LIST})))
endif

And then your build clauses would look mor or less like

$(PROJECT_ROOT)foo.o: $(PROJECT_ROOT)foo.c:
        $(CC) -c -o $@ $<

I sometimes cheat with

r := $(PROJECT_ROOT)

so I can save typing with

$(r)foo.o: $(r)foo.c:

but that's a matter of taste. Be careful about doubled slashes at the
boundary because they can confuse make. I.e. if PROJECT ROOT has a
trailing slash, don't say $(PROJECT_ROOT)/foo.

The above is some combination of accepted modern wisdom and my own
humble opinions. It may not matter at your scale, but doing things
this way makes parallel builds possible and incremental builds
reliable. The fatal flaw of the recursive model is that make never
knows all the facts at the same time.

David Boyce