bug#21218: ls -d

[Bob Proulx]

Sneeh, Eddie wrote:
>  The problem is the behavior of ls -d is so erratic it's difficult to
> describe in a concise way.

This statement pains me to read.  Because it shows that there is still
a misunderstanding on ls and ls -d.

> I haven't seen the code or the spec, so my suggestion is based
> solely on observation (which may or may not be complete).

Let me try to describe it.  In the beginning that we care about was
the Unix file system.  In this file system everything is a file[1].  I
am going to say that again.  Everything is a file.

Some files are special files.  Most files in /dev are special files
such as the block and character devices that immediately people think
about.  But directories are files too!  Directories are simply files
in the Unix file system.  Directories are another example of special
files.

The 'ls' command reads directories, which are special files, and lists
their contents.  The primary purpose of 'ls' is to list directories.
Focus on that task.  Read the directory file.  List it out.  Here is
an example.  In my example I am going to talk classically and simply
and acknowledge that some of the system and library calls have been
modernized but that doesn't matter because the behavior remains the
same.

  ls /

The 'ls' command will have one program argument.  That one program
argument is the "/" string.  The 'ls' program will stat(2) the
argument to see if it exists and if it is a directory.  If it is a
directory then it will treat it specially and call opendir(3) on that
string.  If that succeeds then it reads the directory and lists out
the content.  It only does this for directories.  In this way
directories are treated specially by 'ls'.  If the program argument is
not a identified by stat(2) as a directory then it is simply printed
out normally.

If 'ls' is not given any program arguments then it defaults to listing
the "." directory.  The 'ls' command without arguments:

  ls

Is the same as:

  ls .

This is the same thing.  The 'ls' program opens the "." string and
gets a handle to the file.  It then reads the directory and lists the
contents found there.

What about shell wildcards?

First they are called *shell* wildcards.  They don't have anything to
do with 'ls' at all.  The shell, typically /bin/bash, scans the
command line and *replaces* any shell wildcard with the expansion of
the file glob.  The '*' is called a file glob because it replaces a
glob of characters.  There are also [...]'s too.  This replacement
happens at the command line level and happens before the shell
invocation of 'ls'.

  mkdir test
  cd test
  touch file1.txt file2.txt file3.txt
  ls ./*.txt
  ...the ./*.txt is expanded and replaced by the shell...
  ls file1.txt file2.txt file3.txt
  ...the 'ls' command has three program arguments provided by the shell...
    file1.txt file2.txt file3.txt

The 'ls' program has no way of knowing it was invoked with a shell
wildcard.  The shell replaced that wildcard before ls was launched.
All the 'ls' program knows is that it has a list of program
arguments.  It will walk through each of them in turn and list them
out program argument by program argument.

Now consider this and this next example.

  mkdir dir1.d
  touch dir1.d/file11.txt
  touch dir1.d/file12.txt
  ls -log *.*
    -rw-rw-r-- 1  0 Aug 12 11:33 file1.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 file2.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 file3.txt

    dir1.d:
    total 0
    -rw-rw-r-- 1 0 Aug 12 11:42 file11.txt
    -rw-rw-r-- 1 0 Aug 12 11:46 file12.txt

Not that the files file1.txt, file2.txt file3.txt plus dir1.d were
what was listed.  The files file11.txt and file12.txt are part of the
dir1.d listing.  It is subtle but it is really dir1.d that is being
listed.

That is useful.  But it is also often not desired.  What has happened?
Let's use 'echo' to show what program arguments the the shell is using
to invoke the 'ls' command.

  echo ls *.*
    ls dir1.d file1.txt file2.txt file3.txt

And so here we see that 'ls' has four program arguments.  The dir1.d
argument is a directory.  Therefore 'ls' reads the directory and lists
the contents.  And in doing so the directory itself is skipped from
the listing.  What is the permissions on dir1.d?

  ls -log .
    total 0
    drwxrwxr-x 2 80 Aug 12 11:46 dir1.d
    -rw-rw-r-- 1  0 Aug 12 11:33 file1.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 file2.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 file3.txt

Okay.  The "." directory was listed.  That included the entire
directory.  But I just want to list the dir1.d directory by itself.

  ls -log *.d

Same as just listing the directory.  I just want to see the
permissions on the directory and not the contents.

  ls -log dir1.d
    total 0
    -rw-rw-r-- 1 0 Aug 12 11:42 file11.txt
    -rw-rw-r-- 1 0 Aug 12 11:46 file12.txt

That isn't listing the directory.  Because 'ls' has a program argument
and it is a directory it is listing the contents of the directory.
But I want to list the permissions on the directory itself!

How can this be prevented?  How can we instruct 'ls' that it should
not recurse down into a directory listed as a program argument?  If
only there were some 'ls' option to prevent it treating directories as
special.

There is.  This is the '-d' option.  At least as early as 1979 in V7
the 'ls' command included the '-d' to prevent it treating directories
specially.  If the '-d' option is applied then 'ls' won't treat
program arguments that are directories any different than any other
argument.  The 'ls' command will simply print it for display the same
as any other entry.

  ls -ldog dir1.d
    drwxrwxr-x 2 80 Aug 12 11:46 dir1.d

Which also allows us to use it this way too.

  ls -ldog ./*.*
    drwxrwxr-x 2 80 Aug 12 11:46 ./dir1.d
    -rw-rw-r-- 1  0 Aug 12 11:33 ./file1.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 ./file2.txt
    -rw-rw-r-- 1  0 Aug 12 11:33 ./file3.txt

As a quick aside here I always list wildcards ./* or similarly.
Because otherwise files starting with a '-' option start character
might be confused with being a filename.  With ./* that would be
expanded to be ./- and that will be avoided.  Pedantically I should
use '--' to explicitly say I am done listing options.  But personally
I always use ./* as a personal preference.

  ls -ldog -- *.*

And so we have the '-d' option.  The purpose of which is to list the
any directory arguments themselves and not their contents.  In the V7
docs it described it saying, "If the argument is a directory, list
only its name, not its contents (mostly used with '-l' to get status
on directory)."  The coreutils documentation says:

  ‘-d’
  ‘--directory’
     List just the names of directories, as with other types of files,
     rather than listing their contents.

As you can see this has nothing to do with filtering what is
displayed.  Nor with selecting what is displayed.  The '-d' changes
how 'ls' treats directories listed as program arguments.  If the
program argument is not a directory then it has nothing to do.

This is why when people try 'ls -d *.txt' it is unlikely to ever be
what they want.  It is unlikely that file1.txt would ever be a
directory.  If it isn't a directory then the -d option won't have
anything to do anything in that context.  The caller may be wanting ls
to select based upon the type of the object file but that isn't
related to using the '-d' option.  And specifically selecting files
(finding files) by file type is exactly what 'find' does which is why
everyone always redirects such questions to 'find'.

Bob

P.S.

> Assaf Gordon wrote:
> > It is a long-established behavior of 'ls', one that has been
> > standard on many operating systems and for many years (possibly
> > decades? I'm too new to know for sure).

Try 40 years.  You can read the V7 source and see it there.  After
four decades it truly is long standing behavior. :-)

[1] In Unix everything is a file.  Leading to a joke that in the Linux
kernel everything is a file system.