Re: Makefile idiom for referencing URL sources

[Kaz Kylheku (gmake)]

On 2021-12-16 11:17, Johnson, Richard wrote:
> Hi Alll--
>
> I have a Linux project that builds against URL dependencies.   Make
> itself does not recognize URL resources and  the ':' in the URL naming
> scheme is problematic.

I'd want this to cache the remote materials locally, and check their
cryptographic hashes (SHA256 or whatever).

That pretty much requires a fetch-and-validate layer which can be
entirely opaque to the rest of the build system. That can be some kind
of external tool invoked by make in the right places to manage the
fetching, validation and unpacking.

The fetch system can indicate that things are present in the cache
by touching stamps, which include the object name and expected
version.

Then you can write a make rule like, against our fictitious
fetch tool:


   coolparser/%.c: .fetch-cache/coolparser-$(coolparser_version).stamp
        fetchtool --package coolparser-$(coolparser_version) 
--extract-file $@


   .fetch-cache/%.stamp:
        fetchtool --update $(basename $@)


Suppose that coolparser/lex.o needs to be build, and requires
coolparser/lex.c, which doesn't exist.

That matches the first rule above.


Suppose the .fetch-cache/coolparser-123.stamp doesn't exist.
Then fetchtool --update coolparser-123 is called.  The tool does its job
of fetching everything, validating it and caching it. Then the stamp 
file is
created.

fetchtool reads some file(s) that contains needed information, like
: What is the URL for coolparser, and how does the version number
fit into it? What kind of object is it: tarball, git repo, individual
file, whatever.  What is the SHA256 sum of this object?

Next the first rule fires: the .c file is out of date with respect
to the now present .stamp file. fetchtool is invoked again; this time 
it's told to
extract a file out of coolparser-123. (For simplicity are assuming here 
that there
is no directory shape remapping: the coolparser/lex.c relative path 
makes
sense to the tool, and is the same path we are using in our tree.)

OK, so now we have a coolparser/lex.c in our tree; everything else
proceeds regularly.

You can now work on fetchtool in your favorite scripting language 
instead
of cramming its functionality into make rules that are based on lower
level primitives like wget, sha256sum, file manipulation, etc.