Re: posix command search and execution

[Robert Elz]

    Date:        Tue, 7 Nov 2023 23:04:10 +0100
    From:        Mike Jonkmans <bashbug@jonkmans.nl>
    Message-ID:  <20231107220410.GC27780@jonkmans.nl>


  | In particular, a user function with the name of a standard utility,
  | will be called at this point.

Yes.   That much is very clear.

  | Chet mentioned that. But I find the Austin-discussion hard to read.

It can be, yes.  Particularly when you have to cope with stuff
being discussed in places that aren't public, but affects the results.

  | It makes sense to partition the builtins in three categories with
  | a separate name for each.

That's more or less what has been done now, with the specials, intrinsics,
and regular builtins.

  | Unusuality sketch:
  | - the shell provides a builtin for a standard utility
  | - the distributor provides a function for the same utility

What counts as "the implementation" is always (apparently) an unspecified
issue, splitting things into 2 camps (as almost every standards org
tends to do, there's "us" and "the rest of them") never works well
in practice, yet continues to be done for all kinds of purposes, as it
always seems to provide a nice clean answer - yet as that shows, never
really does, as what is called what always depends upon from what point
of view you're looking, and there tend to be many.

But:
  | in /etc/profile

in that case it would be not be "provided by the implementation" - anything
loaded from profile files is a user add on (even if it is forced upon the
user by their admins) - if the local user doesn't want that function, they
can simply remove it from /etc/profile (or not use that at all).

  | Are scripts in /etc/profile considered part of the implementation?

So, no, I don't think so.

  | It is all kind of theoretical. What wonders me is that the POSIX
  | specifications and definitions sometimes are imprecise or lacking.

They sometimes are, often because no-one considered some weird case,
so didn't think to actually write down what happens then - in the POSIX
world that's called implicitly unspecified.  Occasionally there are
simple errors (minor ones, like spelling, grammar, punctuation or
formatting issues - which are easy to overlook, though annoying) to
major ones where the standard says what someone thought was correct,
but they were mistaken.

But it isn't always the case, even when it seems so - to really
understand it, you really have to read and comprehend (almost) the
whole damn thing (all 4000 approx pages of it).   The text in one
place often says nothing about something which is specified in some
entirely different context.

  | > Yes.   But almost all shells implement it that way, so the
  | > seemingly logical assumption is mostly backed by experience.
  |
  | Again this is not too precise for a standard.

Yes, but the point is that this isn't what the standard says, more like
what it really should say.   This PATH search for builtins stuff is an
invention by people trying to force something upon shells that none of
them implemented, because they thought it would be better.

It is actually messier that it first seems, if you have a builtin command,
it isn't simply finding that command's name in a PATH search that is
required, but that the directory (from PATH) in which it was found be the
one "associated" with the builtin command (how exactly that is determined,
or what it really even means is not specified anywhere).   The effect is
that if a shell with a builtin "test" believes that the associated
directory is /bin and someone's path is /usr/bin:/bin - and there is a
test command in /usr/bin (even if /usr/bin and /bin are the same directory,
or /bin/name and /usr/bin/name are linked and so invoke the same thing)
then the filesystem command is supposed to be invoked rather than the
builtin.    There are reasons for that, but they're really fairly stupid.

[Finding the standard utilities]
  | Good to know. I was wondering about that.
  | But where is that being specified?

Somewhere in the conformance stuff I believe it is, I don't have
the time to go look for it at the minute.

  | I already can't find the definition of standard utilities.

There really isn't one - it is just a shorthand for "utilities
specified by the standard" - less to type (and read).

  | Ok. I was not aware that all standard utilities have to be in PATH.

Not the utilities, or not the way you phrased that, the directories in
which the standard utilities are found have to be in PATH (and PATH needs
to be ordered such that they are found, not some other utility of the
same name).    But I expect that's what you meant.

  | One might want to override `.' (source in bash parlance) for logging.

In a portable script you cannot, as POSIX only requires that "name"s work
as function names, and "." is not a "name" (a "name" is the same syntax as
used for variable names - which doesn't include special or positional params).

Most shells will allow it (POSIX doesn't forbid it) but you cannot rely
upon it working.

  | Can this intrinsic list be amended with any user loaded builtins?

I don't see why not, but the concept of loadable builtins is beyond
the standard, so more or less anything is possible there.

  | That referred to the POSIX utilities link.
  | https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

As I said in my recent reply to Chet (except that his insane practice of
including a Reply-To to just him, excluding the list, which my MUA follows
as instructed by the header unless I remember to override it, meant that
that reply went only to him), that list doesn't include the special builtin
utilities (they come in XCU 2.15), the section just before XCU 3, which the
page from your URL is the index for, and the special builtins are also
standard utilities.

kre