Re: Inconsistent behavior of core utilities

[Kaz Kylheku]

On 2022-08-22 15:01, Dave Close wrote:
> But if I try to simplify that command by inserting the output of the
> ls command above, it doesn't work.
> 
>   $ grep CALL $( ls --quoting-style=shell-escape-always ? )
>   grep: '#': No such file or directory
>   grep: "'": No such file or directory
>   grep: ',': No such file or directory
>   grep: '`': No such file or directory
>   grep: 'A': No such file or directory
>   ...

What you want is:

   grep CALL ? | wc

Never process the output of ls to get a list of names (let alone with
GNU options that mangle the output).

THink about what you're doing. It's the ? syntax doing all the work.
Your $(ls ...) command contains ?, and the shell expands that notation
into file name arguments *before* ls is invoked.

ls isn't doing anything useful there, because the ? syntax can be
applied in grep command.

> Unfortunately, that simple approach doesn't work if I change the ? to
> * because of the special characters in many names. 

If you avoid using ls, * should work fine in the grep command line.
You should pass -- to grep in case some of the names look like command
line options.

   grep -- CALL *

There is a difference in the output of grep pattern * and
grep pattern ./* because the output includes file names,
which are reported as they are given to grep.
 
It does work if
> I use ./*.
> 
>   $ grep CALL ./* | wc
>   16325   61444  950870
>   $ grep CALL $( ls ./* ) | wc
>   16375   61601  953451




> 
> So I almost have a solution. Except the extra 50 lines found in the
> second command are disturbing.

You're telling the shell to take the output of "ls" and capture
it as a string, and then break that string into words on whitespace,
and then perform path name expansions on it and whatnot. 

In other words, the output of ls is partially processed as if it
were shell syntax. It's not a full "eval", but some of it.

For instance if one of your files is named *, then that * will
expand.

Look at this experiment:

  $ mkdir experiment
  experiment $ cd experiment
  experiment $ touch '*' foo
  experiment$ ls -l
  total 0
  -rw-rw-r-- 1 kaz kaz 0 Sep  2 07:57 '*'
  -rw-rw-r-- 1 kaz kaz 0 Sep  2 07:57  foo
  experiment $ echo *
  * foo
  experiment $ echo $(ls *)
  * foo foo

$(ls *) produces the text   * foo,  which, because it isn't
quoted, is split into two fields, and subject to filename
expansion: the * expands into the two fields    * foo,
and so we get   * foo foo.

Quoting won't fix it because then we get a single
field with spaces.

Consider these examples also:

  experiment$ for x in *; do printf "<%s>\n" "$x"; done
  <*>
  <foo>
  experiment$ for x in $(ls *); do printf "<%s>\n" "$x"; done
  <*>
  <foo>
  <foo>
  experiment$ for x in "$(ls *)"; do printf "<%s>\n" "$x"; done
  <*
  foo>

Also:

  experiment$ for x in ./*; do printf "<%s>\n" "$x"; done
  <./*>
  <./foo>

Within a command line, try to avoid inserting any text
processing between the output of file name expansion and
the execution of that command line, not just ls.
If we insert another layer of echo, it will wreck things
the same way:

  experiment$ for x in $(echo *); do printf "<%s>\n" "$x"; done
  <*>
  <foo>
  <foo>
  $ for x in "$(echo *)"; do printf "<%s>\n" "$x"; done
  <* foo>

So forget about ls; but if you see a problem with * misbehaving
somehow due to special characters, you should report that
(to the developers of the shell you're using, not the GNU
Coreutils mailing list.)

Cheers ...