Re: build system rules & algorithms

[Stephan Beal]

On Wed, Jun 10, 2009 at 4:08 PM, Paul Smith<address@hidden> wrote:
> On Tue, 2009-06-09 at 21:08 -0400, Mike Shal wrote:
>> http://gittup.org/tup/build_system_rules_and_algorithms.pdf

Very! The best build-related paper read i've seen since RMCH. (Though
wikipedia also has an interesting page about make.)

> When make was created 30+ years ago, the only way to determine what's
> been changed with any accuracy and reliability is modification times,
> because these are maintained by the operating system and are unlikely to
> be changed "arbitrarily" by hand... at any rate, requiring that the user
> not do so is not an onerous burden since they rarely want to anyway.
> Once this is decided, the current algorithms and behavior of make
> follow: there is no way to know what's been changed without going out
> and looking at each file.

If Make assumes that it is the only process changing files, it might
be able to cache the results somewhere (but where is a whole can of
worms in and of itself, since many projects like read-only source
trees). sqlite3 might actually make a good basis for this type of
thing, since SQL could theoretically be used to calculate build orders
and even output Make code. Granted, make is often not the only process
which is creating files (e.g. the configure script does this, too.),
so this approach might be less useful than it initially sounds.

> Further, make gains a lot of simplicity (but
> at the cost of some performance as you point out) by avoiding the need
> to maintain a separate state:

Aha, i spoke too soon above.

> Going to a Beta build system, on the other hand, requires that some
> entity maintain a highly if not completely accurate accounting of all
> the files that have changed.  That is NOT a simple problem to solve.
> Delegating that problem to IDEs and similar simply ignores the reality
> of development (at least in my world).

IDEs make the assumption that they are the only ones fiddling with the
files. This is arguably most often correct in environments which use
full-fledged IDEs, barring an external source control tool which will
update source files now and then.

>  People don't tend to use IDEs
> much, and when they do they certainly don't do every operation using
> them.

While i don't use an IDE (*cough*xemacs*cough*), most programmers i
speak with do tend to use them. i was recently at a Qt conference with
about 600 or 800 developers, and 90%+ of them were using Mac or
Windows dev environments. Depressing. Then again, they're not using
makefiles directly, either (or their using a front-end which generates
makefiles which they don't touch directly). It's probably true that
90%+ of the people on this list don't use IDEs (and thus their
interest in the intricacies of Make).

> Any discrepancy in the list of modified files results in incorrect
> results of a Beta build, and possibly in very subtle and hard to
> discover ways.

Also my opinion. On a related note, my major gripe with Alpha builds:
i come from the school of thought which (strongly) believes that a
change to a Makefile requires a rebuild of all targets controlled by
the Makefile, for the simple reason that the Makefile controls all
options sent to the compiler. If the Makefile changes, compile
flags/options might have changed, and not doing a full rebuild can
lead to difficult-to-find bugs which mysteriously disappear when "make
clean; make all" is run (because a -DFOO=3 was changed to -DFOO=4). In
an Alpha build this school of thought suffers considerably unless the
whole project builds in a trivial amount of time (say, less than 10
seconds).

> I really do like the ideas in your paper and I think it's a worthwhile
> exercise to skip over the very hard initial problem (knowing "what's
> changed") to see what we could do if we knew that: whether it's even
> worthwhile to try to solve it.  I think your paper shows very nicely
> that we could do a lot better if we can solve that problem, so it's
> worth thinking about those solutions.

Amen!

While RMCH makes some interesting points, i'm of the opinion that a
properly constructed recursive make is still the best general-purpose
way to structure most trees. (Let's not forget that "general purpose"
anything will always outperform specialized solutions for particular
cases.) Once a top-level build is done, subdirs can be built very
quickly, and trees can (in my experience) be structured so that the
build makes it impossible to have circular references between modules
in the source tree (as they are as problematic to the build process as
they are philosophically troubling). i'll spare you all my long-winded
evangelism, but this topic is one i can certainly go on about for a
while.

:)

Thanks, Mike :).
-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/