Makefile idiom for referencing URL sources

[Johnson, Richard]

Hi Alll--

I have a Linux project that builds against URL dependencies.   Make itself does 
not recognize URL resources and  the ':' in the URL naming scheme is 
problematic.  I was able to sidestep these constraints with the following idiom 
using timestamp files and lexical substitution.   The lexical operations 
transforms the URL scheme to a local file path

It seems to work -- at least when you have a reasonably modern HTTP server such 
as Apache or nginx
I was wondering if anyone else had experience with a similar scheme.


# Define tracking files for URL sources. These are empty files with a 
modification date
# set to the Last-Modified attribute of the source.   Tracking files reside in 
a ".<scheme>"
# directory with a pattern matching the resource and path.
#
# The use of ":" for scheme and port separator conflicts with Makefile syntax.  
This can
# be accomodated with $(substr <scheme>:,.<scheme>,<url>).  Port specification 
is not
# currently supported.
#
# Dependencies: curl, wget, touch
# Requires: HTTP headers contain "Last-Modified:" field.
.http/%:
        #...establish timestamp of the source
        mkdir -p $$(dirname $@)
        tstamp="$$(curl -sI $(subst .http,http:,$@) 2>/dev/null | sed -n 
's/Last-Modified: //p')" && \
        touch -d "$${tstamp}" $@

URL_SOURCES: http://some-url ....
URL_TRACKING : $(subst http:,.http, $(URL_SOURCES))

# Pattern for single file retrieval
$(DESTPATH)/% : $(subst http:,.http, http://some-url ...)/%
mkdir -p $(dir $@)
        curl $(subst .http,http:,$^) > $@

# Pattern for directory tree retrieval using wget.
# Note: The --cut-dirs parameter is dependent on the source path
$(DEST_DIR) : $(subst http:,.http, http://some-url)/
mkdir -p $(dir $@)
        wget -r -np -nH --cut-dirs=n -P $(dir $@) $(subst .http,http:,$^)