Re: new feature idea: ingesting processed rulesets

[David Boyce]

> DIR := $(shell pwd)

My only contribution to this is to point out that I believe the above construct, while very common, is unnecessary. The $(CURDIR) variable is defined by the manual to be an absolute path to the current working directory of the make process. Combined with the fact that make has no way to change directories (recipes can cd but make itself cannot, which is a feature), I see no way that invoking a shell and pwd can improve on simply using $(CURDIR).

(Other) David

On Thu, Jun 22, 2023 at 5:13 PM David A. Wheeler <dwheeler@dwheeler.com> wrote:

On Sun, 2023-06-11 at 12:29 +0000, Zoltán Turányi wrote:
>> Couldn’t these be handled by creating a "namespace" concept for
>> rules? One could merge 2 Makefiles by prefixing each target with the
>> directory of the Makefile. This would make dir1/all different from
>> dir2/all. Also in recipes variables, pattern rules could be used from
>> the Makefile describing the recipe. Would this not be possible (in
>> theory, at least)?
>

On Jun 22, 2023, at 2:41 PM, Paul Smith <psmith@gnu.org> wrote:
> Sure, but obviously it's not sufficient because some makefile rules DO
> want to refer targets in other makefiles / directories.  I want my
> target to depend on the "libfoo.a" target from some other directory.
> How does that work?

If you want to create multiple makefiles and specify a "current namespace for files
in a directory's makefile", and you're using GNU make,
that's easily done by using this construct in the makefile fragment in each directory:
DIR := $(shell pwd)

Below is a sample of what this would look like. I'm using ".mk" as the extension
for fragments. A real makefile should generate the dependencies, not hand-jam them,
but hopefully you get the idea.

If you *need* portability with BSD make as well, you could use this instead:
DIR != pwd
... but if any directory includes "$" you're in trouble :-).
POSIX make is very minimal & it's usually better to just say "must use GNU make".

If you always run "make" from the root directory, this should work fine.
It should take less than a second for the no-work case even in larger systems
(e.g., 100K files or so being managed as dependencies).
A user *could* run "make" in just a subdirectory as long as there are
never cross-directory dependencies, using this approach.
However, it will end BADLY if there were EVER
cross-directory dependencies AND a user ran "make" using only the
makefile in a *subdirectory*. In my experience, the cross-directory
dependencies WILL eventually happen. So it's better to NEVER name them "makefile"
(so users won't accidentally screw up building),
and instead name these makefile fragments with a ".mk" extension.
Create your makefiles in each directory, and run "make" at the top directory,
where its "makefile" transitively includes the others.

If it takes more than than a second to run "do-nothing" at the top dir and you have <100K files,
then there's a serious problem that needs fixing.

If you think you *have* to run make in
a way that it can *only* see the dependencies of a subdirectory,
I think you're optimizing the wrong problem.
Make is *very* fast at parsing makefiles & at checking file timestamps.
But if you give Make the wrong data, it will do the wrong thing.
You may then try to "fix" it by overspecifying things, leading to
complex & unmaintainable makefiles that will be slow & wrong.

--- David A. Wheeler

~~~~~~~~~~~~~~~~~~~~~~~~~~~

===
./makefile
===
# Demo getting current directory for use in file references

DIR := $(shell pwd)

$(DIR)/result: $(DIR)/result.o $(DIR)/subdir/bar.o

clean:
rm -f *.o */*.o

include subdir/fragment.mk

===
./subdir/fragment.mk
===

DIR := $(shell pwd)

$(DIR)/bar.o: $(DIR)/bar.c

===
./subdir/bar.c
===
#include <stdio.h>

void bar(void) {
    printf("In bar.\n");
}

===
./result.c
===
#include <stdio.h>

extern void bar(void);

int main(void) {
    printf("In main.\n");
    bar();
    return 0;
}


>
>> My problem is that contrary to the make wisdom of writing a single
>> Makefile (to which I agree) most projects are still divided into
>> parts with separate build definitions. One can debate if this is good
>> or bad - for me it is a requirement to handle this case efficiently.
>> (If you do not agree there is no point in discussing further.)
>
> I wouldn't say "requirement", especially in conjunction with
> "efficiently".  If people want "more efficient" they may have to be
> willing to pay the up-front cost of changing their makefiles and I
> think that's a defensible position.
>
> But you are basically saying what I said in my previous email: the only
> way it makes sense to try to implement something like this in GNU Make,
> is if it could be done in such a way that people DON'T have to modify
> their existing makefiles.
>
> Because if they have to modify their makefiles anyway they might as
> well just do the work to make them non-recursive in the first place.
>
>>> However, I think this will be extremely difficult, because makefile
>>> targets are so free-form and prerequisites and targets are not
>>> actually necessary files at all.
>>
>
> Also, how can you replace these target references if they appear inside
> a recipe?  Make can't parse shell script content in recipes (if they
> are even shell scripts at all; SHELL could be /usr/bin/perl or
> something).  You can't just go through all the recipe text and replace
> instances of the target "foo" with "dir1/foo".
>
> We couldn't use a simple namespace naming convention like you suggest
> of prefixing the directory because targets in the current makefile
> could already be named "dir1/all" for example.  So you have to find a
> way to talk about namespaces which don't conflict with existing naming.
>
> Also what about things like make recipes that invoke recursive make,
> _inside the same directory_.  Automake-generated makefiles do this all
> the time.  If the namespace is based solely on the containing directory
> is that sufficient?  Often the recursive invocation of make overrides
> variables on the command line, so it seems like using the same
> "namespace" wouldn't work.
>
> And remember the premise here is that all these things have to work
> WITHOUT any changes to makefiles (or Makefile.am files for automake)
> themselves.
>
> I'm not saying I know for a fact that it couldn't be done.  But it
> definitely seems to me like there are lots of issues that have to be
> considered and worked out, and I'm not really sure that it would really
> be that much more efficient anyway.  The real efficiency gain would be
> from:
>
> (a) Not invoking multiple make processes at all... but the proposal on
> the table is that we'd still invoke the make process, and it would
> still parse its makefile, it would just ship the post-parsed content
> back to the parent make, so that's at best equivalently efficient and
> almost certainly slightly slower; and
>
> (b) Allowing make to have a "global view" of all the rules and targets
> so it can more efficiently run things in parallel... but here we're
> talking about creating multiple namespaces anyway so can we really have
> these multiple namespaces to separate target AND have this global view
> of targets for better parallelism at the same time?
>
> Probably what needs to happen to help answer these questions is for
> someone (but, not me :)) to take a sample automake-enabled software
> package, look at the makefiles generated by automake for that package,
> and come up with a concrete proposal for how it might work and some
> comments on how the result would be more efficient.
>