Re: hidden dependencies

[Krzysztof Cieniuch]

Philip Guenther wrote:
> On Thu, Jan 20, 2011 at 2:35 AM, Krzysztof Cieniuch
> <address@hidden> wrote:
> ...
>   
> In that previous project, each module generated either a library (with
> matching header(s)) or a program.  The Makefile for a library recorded
> the include path for accessing the headers it supplied, while the
> Makefile for any module that depended on other modules' libraries had
> to declare which modules it depended on, so that it could both get
> access to the depended upon modules' include files (via their recorded
> include paths)...and so that the Makefile system could declare the
> order-only dependencies involving all the involved generated header
> files
>
>
>   
> > My build system includes only necessary makefiles when building project
> > module (subdirectory)
> > Parsing each time 500 makefiles is not an option since it takes to long (on
> > Sun T5440 it takes around 2.5 minute) and if developer is working
> > on single module edit/recompile cycle would take to long.
> >     
>
> (2.5 min?  Wow, T5440's are even suckier than the hardware that
> project had to deal with!)
>
>   
Yeah they are terrible developer build boxes.
For comparison parsing makefiles for the same project on xeon takes 11 
seconds. and on Itanium
with roughly the same clock 28 seconds.
But to be fair it is not that bad when building from scratch it takes 
roughly the same
time to compile it has 256 virtual cores it is real pleasure to watch  
make -j256 in work :-)))
Although it took a while to develop tools that would put back build output
as you would invoke single threaded recursive make build system

> If each directory declares the directories that it has a dependency
> on, then you could restrict your Makefile inclusions to those that are
> necessary for the (active) targets.  It does get a bit tricky to get
> right, I'll admit.  Once your Makefile setup gets complex enough it
> requires testing and debugging like any other program...
>
>   
My build system works exactly as you described there is set of core 
makefiles
the build system "brain" and few hundred project makefiles which are not
really makefiles but more like build directives.
Example makefile in module that builds shared library looks like this:

include mk/init.make
TARGETS:=t1
t1_INCLUDES:=include/dir
t1_LIBDEPS:=lib1 lib2
t1_SRCS:=file.cc
t1_TARGET_NAME:=utils
t1_TARGET_TYPE:=shlib
include mk/make.make

where LIBDEPS lists not real library names but resources ids.
So if  dependencies location in project changes it is just a matter to 
updating resource definition.
Also there is  feature to allow "import" selected resources from one 
project to other.

There is a lot of different features as well :
you may have multiple targets in one Makefile.
you may mix c++/c with idl and rpc definition sources like this:
t1_SRCS:=file1.cc file2.c file3.idl file4.x
you may set compile options per file e.g.
t1_SRCS:=[no_optimize]file1.cc [obj:file1_c]file1.c
(by default object file names will be derived from source file but in 
above sample
we would have name clash so file1.c will be compiled into file1_c.o )
There is global INSTALL_FILES directive for easy resources copying
INSTALL_FILES:=file1 [chmod:755]                          $(INSTALL_DIR) \
                                file2 [chmod:644,mv:file2_copy]  
$(INSTALL_DIR) \
                                file3 [ln:file3]                       
        $(INSTALL_DIR)
Automatic dependency generation including command arguments
i.e. if command to build target  changes the target is rebuild
and so an and on ...  (enough showing off :-))

Each time developer makes a build only necessary makefiles are included 
and parsed so make logically 'sees' only one makefile.
Since all rules are generated this is quite intensive process. But there 
is caching and debugging  mechanism that allows
to print generated rules instead of invoking them i.e. you may direct 
build system so instead rule evaluating it prints them using $(info ).
So it easy to verify if generated rules are correct or troubleshoot why 
something doesn't work as expected.
Note this is different from -n switch since that debug output is fully 
valid makefile with all the generated recipes.
It can be used to build project this is how caching works. The size of 
cached makefile in project root is about 11M
and then even on T5440 it takes 20 seconds to parse that makefile 
instead of 2.5 minute.

I think it would be worth to put some good ideas or best practices into 
make manual instead of
chapter "5.7 Recursive Use of make"  :-)

Krzysztof