Re: Idea: Add .COMMANDCHANGE and .CACHE

[David A. Wheeler]

On Tue, 11 Jun 2019 14:25:10 -0800, Britton Kerin <address@hidden> wrote:
> > I think .COMMANDCHANGE is not complex.
> > Since it caches the expanded command, at worst it will re-run a script
> > when before it would not have done so.
> > So it's relatively low risk; at worst, it'll run a command more often than 
> > before.
> > It won't rerun if only an environment variable changes, but that is 
> > *already* true.
> 
> True but it's a sort of misleading way to look at things.  With .COMMANDCHANGE
> you're effectively saying "add this thing and you can efficiently intermingle
> build system changes with other development without worry".  Fail to deliver 
> on
> that promise and the feature will likely cause net pain.

True, but it's hard to image what could be missing.
This is already *WAY* more specific than what make normally does.

By default make doesn't consider the modification date of a script,
you have to even add THAT to the list of prerequisites.  When it's added to the
list of prerequisites, make *only* considers the modification time of the 
script,
and does NOT consider if the parameters used to call it
from the makefile (so if the parameters make change, it won't rerun).

.COMMANDCHANGE, by contrast, is FAR more conservative
than the typical use of commands as prerequisites.
If the any text in the script changes, or if anything changes in how its
parameters are expanded is changed, it will force a rerun.

If a key underlying environment variable changes then yes, it will be missed,
but that is *already* true in make, and at least in COMMANDCHANGE
you can include the environment variable in the key's data source.

I can't imagine what would be missing from .COMMANDCHANGE
that "make" normally considers.  Any other argument would already
be true for make as it is.

The argument changes for .CACHE of course.
In that, the failure mode is a bigger deal & there's a possibility that
something has been missed.  So let's discuss!!


> I'm unlikely to ever use &, because I predate it by decades and  
> stamps/proxies
> have always worked fine for me (I've never been tempted to edit or delete a
> proxied file and been burned as a result).  So it would be nice if
> .COMMANDCHANGE/.CACHE still worked without &, and I think to be really correct
> it must since nothing in existing make forbids these common arrangements.

.COMMANDCHANGE will work just fine if you don't use "&:"
and just use ":". For example, stamps and proxies will work very nicely.
For example, if you say:
~~~~
BB CC: .COMMANDCHANGE DD EE
<TAB>command
~~~~
Then for BB and CC there will be *separate* hashes that record
the command run for them.

The issue with .CACHE is more complicated.
Any caching system needs to know:
(1) what inputs could affect the output, and
(2) what the outputs are.

When determining the outputs (#2): if there's one target
*OR* there are multiple targets expressed using "&:",
then the answer is easy.

The bigger problem is when there are multiple outputs
AND they are not listed in a single rule.  Detecting when someone
did "BB CC: ..." and expected it to mean "BB CC &:" will prevent
*that* mistake.  If the user switches to "&:" then no issue.

BUT: If the user uses an intermediate marker, then maybe we
need to do more.  It's conventional use this construct
to indicate when multiple outputs are generated from a single command:

~~~~
BB CC: marker ;
maker: prerequisites
<TAB>command
~~~~

If we naively ran the "marker" rule with .CACHE, then
the cache would only have the marker - not BB or CC. Whups.

One solution comes to mind: when "make"ing a ":" rule
with a single target "marker", look for all the rules that depend ONLY
on that marker (ignoring special targets) and have a
present-but-empty rule.  Their targets are always considered
as the additional outputs of creating marker
when creating a .CACHE.  It's an odd rule, but I think it
solves the problem with generality.  Traditionally this is how
"multiple outputs from a single recipe" is represented, so
to cache the outputs we need to detect that traditional representation.

> To be conservative and automatic I would probably include the entire
> environment and then explicitly subtract out e.g. X/DBUS garbage if feature 
> was
> provided to do so.  I guess this could be done with text functions easily 
> enough
> too though.

I imagine we could have some sort of setting that did that, i.e.,
"include everything in environment EXCEPT these environment variables".

> Again, I think what you really want is a key derived from all the recipes in
> the entire dependency DAG of the target.  I think this picks up the case 
> you're
> considering here, plus others where significant side effects exist that aren't
> explicitly mentioned in the DAG at all.

I think that's way too conservative.  If the inputs are the same, and the
command produces the same outputs from the same inputs, that should
be enough.  The dependency DAG is already captured in the dependency tree
that make already processes.

--- David A. Wheeler