Re: Quoting special characters (was: Re: Possible solution for special characters in makefile paths)

[Frank Heckenbach]

Eli Zaretskii wrote:

> > > If we know that a variable was quoted, we will treat it as a single
> > > word.  So I don't really see any serious problems here (of course,
> > > this special handling of quoted strings will have to be coded).
> > 
> > Unfortunately, it's not so easy. In my example above, the value
> > could neither be words-separated-by-spaces like now (would become 4
> > words), nor a single word as you suggest.
> 
> This is only a problem because you want to solve the quoting issue at
> the same time.  If you remove that requirement, the problem goes away.

But I don't like to remove that requirement. :) Really, it's nice if
make can handle filenames with spaces internally, but that's just
half of the way since if you have such filenames as targets or
dependencies, you usually want to do something with them, i.e. use
them in the command line. Take this example:

FILES = $(wildcard *)

target: $(FILES)
        cat $(FILES) > $@

Obviously, the intention of that rule is to concat all files to the
target file. (Don't care whether that's a useful command on its own;
in reality it might create a library from selected files or
whatever.)

Assume there are two files, "foo bar" and "ba\z qu'x". What
does/should happen here?

- Currently, make splits the filenames into words at spaces, so
  FILES contains 4 words ("foo bar ba\z qu'x") which already lost
  information and doesn't and cannot work as intended. That's of
  course the basic problem Paul is trying to solve.

- With Paul's proposals (and mine, which only differ in the internal
  encoding which as you say doesn't matter at this point) FILES
  would contain something like "fooSEPbar ba\zSEPqu'x" which at
  least preserves all information and can be made to work correctly
  in the target list because make will know to decode the string
  there.

  Now the question is how to get this value into a command-line
  correctly. Actually, it is possible with make's existing
  functions, given that it handles encoded spaces as
  non-word-separators as planned:

  Q=$(addsuffix ',$(addprefix ',$(subst ','\'',$(1))))

  This would first escape embedded single quotes, then single-quote
  each word, resulting in "'fooSEPbar' 'ba\zSEPqu'\''x'" which is
  suitable for the shell.

  Of course, there are many practical disadvantages: It's quite a
  mouthful, very unintuitive and easy to get wrong, and when writing
  generic rules (where you don't know in advance which filenames may
  contain spaces) you'd need to apply it to each expansion of a
  variable, including $@, $<, $^ etc. So typical simple command like
    $(COMPILE.o) -c $< -o $@
  becomes
    $(COMPILE.o) -c $(call Q,$<) -o $(call Q,$@)
  which means calling 6 make functions. And implicit rules wouldn't
  use it, meaning they'd be useless for filenames with spaces.

  That's why I suggested SHELL_QUOTE. It's still a bit ugly, but
  less so (if it's automatically applied per word, we don't need the
  addsuffix/addprefix part), and it wouldn't clutter the recipies
  which could still be written as:
    $(COMPILE.o) -c $< -o $@
  This removes one source of errors, and works with implicit rules.

- With your proposal, I'm still not sure what FILES should actually
  contain in this case. You mentioned the case where the filename with
  spaces occurs explicitly in the Makefile, but what about e.g.
  $(wildcard) expansions. For consistency, I assume you'd have make
  insert backslashes or quotes there automatically. (If not, please
  clarify.) So then, FILES would contain e.g. "foo\ bar ba\\z\ qu\'x"
  (note how the backslash in the filename is duplicated to avoid
  ambiguity; also I've escaped the existing quote, so the whole quote
  string is compatible with shell quoting rules, as you suggest).

  So far, I think we can agree. But we seem to disagree what happens
  when such a variable is used in the command-line. I would want to
  substitute it including quotes and escapes, so the full
  information is preserved to the shell, so
    cat $(FILES) > $@
  becomes
    cat foo\ bar ba\\z\ qu\'x > target
  which is fine.

  But from your answer to Paul's example, that's apparently not what
  you want -- or seem to want; I actually think that's due to a
  quirk in the example as I'll explain.

  So let me amend his example to this case to get the full glory.
  Again, let's assume we have two files, "foo bar" and "ba\z qu'x".
  Then what should this print:

    FOO = $(wildcard *)
    foo: ; @echo '"$(FOO)"' '"$(addsuffix .txt,$(FOO))"'

  What I'd like to see is substitution including escapes:
    foo: ; @echo '"foo\ bar ba\\z\ qu\'x"' '"foo\ bar.txt ba\\z\ qu\'x.txt"'
  which prints
    "foo\ bar ba\\z\ qu\'x" "foo\ bar.txt ba\\z\ qu\'x.txt"

  This looks ugly which is probably why you don't want it. However,
  this is just due to the additional manual quoting in the example.
  In an actual rule, you can't quote manually like this in a useful
  way if you don't know which special characters the variable may
  contain, and with full substitution you wouldn't have to, so you'd
  rather write a rule like this:
    foo: ; @echo $(FOO) $(addsuffix .txt,$(FOO))
  which would expand to
    foo: ; @echo foo\ bar ba\\z\ qu\'x foo\ bar.txt ba\\z\ qu\'x.txt
  and print
    foo bar ba\z qu'x foo bar.txt ba\z qu'x.txt
  which shows the real filenames. A command like "cat" in place of
  "echo" would actually see 4 filenames ("foo bar", "ba\z qu'x",
  "foo bar.txt", "ba\z qu'x.txt") and work correctly.

  So if done like this, it might just work as far as I'm concerned.
  I don't know, though, if this will break backward-compatibility or
  POSIX requirements (at first glance it seems it might not, at
  least for Paul's example), or how hard is it to implement since
  make would need to do shell-style un-/escaping with all its
  complexity in various places, but if it's feasible it might indeed
  be the most transparent solution to the user (of a POSIX shell :).