Re: new feature idea: ingesting processed rulesets

[David A. Wheeler]

On Sun, 2023-06-11 at 12:29 +0000, Zoltán Turányi wrote:
>> Couldn’t these be handled by creating a "namespace" concept for
>> rules? One could merge 2 Makefiles by prefixing each target with the
>> directory of the Makefile. This would make dir1/all different from
>> dir2/all. Also in recipes variables, pattern rules could be used from
>> the Makefile describing the recipe. Would this not be possible (in
>> theory, at least)?
> 

On Jun 22, 2023, at 2:41 PM, Paul Smith <psmith@gnu.org> wrote:
> Sure, but obviously it's not sufficient because some makefile rules DO
> want to refer targets in other makefiles / directories.  I want my
> target to depend on the "libfoo.a" target from some other directory. 
> How does that work?

If you want to create multiple makefiles and specify a "current namespace for 
files
in a directory's makefile", and you're using GNU make,
that's easily done by using this construct in the makefile fragment in each 
directory:
DIR := $(shell pwd)

Below is a sample of what this would look like. I'm using ".mk" as the extension
for fragments. A real makefile should generate the dependencies, not hand-jam 
them,
but hopefully you get the idea.

If you *need* portability with BSD make as well, you could use this instead:
DIR != pwd
... but if any directory includes "$" you're in trouble :-).
POSIX make is very minimal & it's usually better to just say "must use GNU 
make".

If you always run "make" from the root directory, this should work fine.
It should take less than a second for the no-work case even in larger systems
(e.g., 100K files or so being managed as dependencies).
A user *could* run "make" in just a subdirectory as long as there are
never cross-directory dependencies, using this approach.
However, it will end BADLY if there were EVER
cross-directory dependencies AND a user ran "make" using only the
makefile in a *subdirectory*. In my experience, the cross-directory
dependencies WILL eventually happen. So it's better to NEVER name them 
"makefile"
(so users won't accidentally screw up building),
and instead name these makefile fragments with a ".mk" extension.
Create your makefiles in each directory, and run "make" at the top directory,
where its "makefile" transitively includes the others.

If it takes more than than a second to run "do-nothing" at the top dir and you 
have <100K files,
then there's a serious problem that needs fixing.

If you think you *have* to run make in
a way that it can *only* see the dependencies of a subdirectory,
I think you're optimizing the wrong problem.
Make is *very* fast at parsing makefiles & at checking file timestamps.
But if you give Make the wrong data, it will do the wrong thing.
You may then try to "fix" it by overspecifying things, leading to
complex & unmaintainable makefiles that will be slow & wrong.

--- David A. Wheeler

~~~~~~~~~~~~~~~~~~~~~~~~~~~

===
./makefile
===
# Demo getting current directory for use in file references

DIR := $(shell pwd)

$(DIR)/result: $(DIR)/result.o $(DIR)/subdir/bar.o

clean:
rm -f *.o */*.o

include subdir/fragment.mk

===
./subdir/fragment.mk
===

DIR := $(shell pwd)

$(DIR)/bar.o: $(DIR)/bar.c

===
./subdir/bar.c
===
#include <stdio.h>

void bar(void) {
    printf("In bar.\n");
}

===
./result.c
===
#include <stdio.h>

extern void bar(void);

int main(void) {
    printf("In main.\n");
    bar();
    return 0;
}


> 
>> My problem is that contrary to the make wisdom of writing a single
>> Makefile (to which I agree) most projects are still divided into
>> parts with separate build definitions. One can debate if this is good
>> or bad - for me it is a requirement to handle this case efficiently.
>> (If you do not agree there is no point in discussing further.)
> 
> I wouldn't say "requirement", especially in conjunction with
> "efficiently".  If people want "more efficient" they may have to be
> willing to pay the up-front cost of changing their makefiles and I
> think that's a defensible position.
> 
> But you are basically saying what I said in my previous email: the only
> way it makes sense to try to implement something like this in GNU Make,
> is if it could be done in such a way that people DON'T have to modify
> their existing makefiles.
> 
> Because if they have to modify their makefiles anyway they might as
> well just do the work to make them non-recursive in the first place.
> 
>>> However, I think this will be extremely difficult, because makefile
>>> targets are so free-form and prerequisites and targets are not
>>> actually necessary files at all.
>> 
> 
> Also, how can you replace these target references if they appear inside
> a recipe?  Make can't parse shell script content in recipes (if they
> are even shell scripts at all; SHELL could be /usr/bin/perl or
> something).  You can't just go through all the recipe text and replace
> instances of the target "foo" with "dir1/foo".
> 
> We couldn't use a simple namespace naming convention like you suggest
> of prefixing the directory because targets in the current makefile
> could already be named "dir1/all" for example.  So you have to find a
> way to talk about namespaces which don't conflict with existing naming.
> 
> Also what about things like make recipes that invoke recursive make,
> _inside the same directory_.  Automake-generated makefiles do this all
> the time.  If the namespace is based solely on the containing directory
> is that sufficient?  Often the recursive invocation of make overrides
> variables on the command line, so it seems like using the same
> "namespace" wouldn't work.
> 
> And remember the premise here is that all these things have to work
> WITHOUT any changes to makefiles (or Makefile.am files for automake)
> themselves.
> 
> I'm not saying I know for a fact that it couldn't be done.  But it
> definitely seems to me like there are lots of issues that have to be
> considered and worked out, and I'm not really sure that it would really
> be that much more efficient anyway.  The real efficiency gain would be
> from:
> 
> (a) Not invoking multiple make processes at all... but the proposal on
> the table is that we'd still invoke the make process, and it would
> still parse its makefile, it would just ship the post-parsed content
> back to the parent make, so that's at best equivalently efficient and
> almost certainly slightly slower; and
> 
> (b) Allowing make to have a "global view" of all the rules and targets
> so it can more efficiently run things in parallel... but here we're
> talking about creating multiple namespaces anyway so can we really have
> these multiple namespaces to separate target AND have this global view
> of targets for better parallelism at the same time?
> 
> Probably what needs to happen to help answer these questions is for
> someone (but, not me :)) to take a sample automake-enabled software
> package, look at the makefiles generated by automake for that package,
> and come up with a concrete proposal for how it might work and some
> comments on how the result would be more efficient.
>