Re: rm -r sometimes produces errors under NFS

[Jim Meyering]

Vincent Lefevre <address@hidden> wrote:
> On 2007-03-07 01:13:11 +0100, Jim Meyering wrote:
>> You want to ignore only certain ENOENT errors.
>> With the current implementation, knowing which
>> to ignore would require recording which
>> names have been successfully unlinked.
>
> In fact I want to ignore all ENOENT errors, but you disagreed.
>
>> >> > Wrong. In the recurse phase, if rm tries to unlink a file, this means
>> >> > that the file has existed. So, this wouldn't be contrary to POSIX.
>> >>
>> >> Your conclusion is invalid.
>> >> What if some other process removed it first?
>> >
>> > AFAIK, POSIX doesn't say that there should be an error in this
>> > particular case,
>>
>> Yes, it does.
>> This part of the standard is clear: "If this fails for any reason ..."
>
> This is not a failure: rm has seen the file and the file has been
> removed.

If you want to discuss further whether rm -r should ignore ENOENT
unconditionally, I suggest that you raise the issue with the people
who work on the POSIX standard (use the austin-group-l mailing list;
subscribe at http://www.opengroup.org/austin/lists.html).  If you can
convince them that you're right, then you'll have a much easier time
convincing me to change the corresponding bits in coreutils.

>> > and IMHO, it is better and more consistent *not*
>> > to return an error. Indeed, consider the following case:
>>
>> Doing that would be inconsistent with other implementations, too.
>
> Do the other implementations do rewinddir too?

You can check, but it's not relevant.
This is independent of the rewinddir vs. NFS issue.

>> Try to imagine why -f has the ENOENT-ignoring semantics.
>>
>> > 1. "rm -r dir" is started.
>> > 2. A second process removes some file in some subdirectory of dir,
>> >    and the "rm -r" process hasn't had the time to see it.
>> > 3. "rm -r" terminates (without any error).
>> >
>> > Why would you want "rm -r" to return an error if some file is removed
>> > by a second process between the time "rm -r" does the readdir and the
>> > time unlink is performed on this file by "rm -r", but have no errors
>> > in the case I've described above?
>>
>> To know that your rm was competing with another file-removing process.
>
> Except in particular cases, one can't know that with the current rm
> implementation. So, this is not a valid reason.
>
>> To conform with POSIX.
>
> This is a point on which I disagree.

If you have more POSIX interpretation questions, please
take it up with the austin-group-l list.

>> To be consistent with all other rm implementations.
>
> See the above question about rewinddir.

As I said above, this is independent of rewinddir.
coreutils' rm -r will never be changed to ignore ENOENT unconditionally.

>> > Also note that in the NFS case, the errors are due to the rewind, but
>> > Point 2c for rm in POSIX is[*]:
>> >
>> >   For each entry contained in file, other than dot or dot-dot, the
>> >   four steps listed here (1 to 4) shall be taken with the entry as
>> >   if it were a file operand. The rm utility shall not traverse
>> >   directories by following symbolic links into other parts of the
>> >   hierarchy, but shall remove the links themselves.
>> >
>> > [*] http://www.opengroup.org/onlinepubs/009695399/utilities/rm.html
>> > (I don't know if this is the latest version...)
>> >
>> > So, if you want a strict interpretation of "For each entry", I don't
>> > think a rewind is allowed.
>>
>> We've come full circle (maybe twice :-).
>> It comes down to having consistent results from unlink and readdir.
>> When readdir returns NULL with no error, I can safely call rewinddir
>> and then use readdir to iterate through any new entries.
>
> But POSIX just says "For each entry contained in file". It doesn't
> say "While there is an entry in file". With rewinddir, you're doing
> "While there is an entry in file", and "rm -r" can loop indefinitely
> (depending on the other processes).

Precisely.
That is how I chose (and choose) to implement rm.
Certainly, there are contrived scenarios in which
GNU rm will not terminate (you'd need multiple concurrent processes
creating entries in a directory that rm is removing, since unlinking is
faster than creation) but one can also argue that rm is doing what the
user wants in that case.

If you'd like to continue with this topic, please take it up with the
Austin Group mailing list.

>> Your system violates that assumption.
>
> NFS violates many things.
>
>> If any unlink fails, rm does not try to call rmdir on any
>> parent directory.  If your case (all unlinks failed with
>> ENOENT) were common, it might be worth handling it by attempting
>> the rmdir if that's the only type of failure.
>
> There is another case. Say, you have two shells 1 and 2:
>
> 1> mkdir dir
> 1> touch dir/a
> 1> touch dir/b
> 1> rm -ri dir
> rm: descend into directory `dir'? y
> rm: remove regular empty file `dir/a'? n
> rm: remove regular empty file `dir/b'?
>
> Before answering, in the second shell:
>
> 2> rm dir/a
>
> Then back to the first shell:
>
> rm: remove regular empty file `dir/b'? y
> 1> test -d dir && echo OK
> OK
>
> IMHO, this is incorrect. Step 4 just says: "If the current file is a
> directory, rm shall perform actions equivalent to the rmdir() function
> defined in the System Interfaces volume of IEEE Std 1003.1-2001 called
> with a pathname of the current file used as the path argument." Nothing
> about the answers for the files contained in the directory.

I understand your desire, but consider this case:
Given a hierarchy d/d/d/d/d/d/d/F where F cannot be removed,
what should "rm -fr d" report?

a single line diagnostic, as it currently does:

  rm: cannot remove `d/d/d/d/d/d/d/F': Permission denied

or one for every parent directory for which rmdir fails
with e.g., ENOTEMPTY?

  rm: cannot remove `d/d/d/d/d/d/d/F': Permission denied
  rm: cannot remove `d/d/d/d/d/d/d': Directory not empty
  rm: cannot remove `d/d/d/d/d/d': Directory not empty
  rm: cannot remove `d/d/d/d/d': Directory not empty
  rm: cannot remove `d/d/d/d': Directory not empty
  rm: cannot remove `d/d/d': Directory not empty
  rm: cannot remove `d/d': Directory not empty
  rm: cannot remove `d': Directory not empty

I want the former.  The "Directory not empty" diagnostics are not useful.

FYI, this discussion has been unproductive, so I am unlikely to continue it.