Re: Threaded versions of cp, mv, ls for high latency / parallel filesystems?

[Phillip Susi]

James Youngman wrote:
> This version should be race-free:
>
> find -type f -print0 |
>      xargs -0 -n 8 --max-procs=16 md5sum >> ~/md5sums 2>&1
>
> I think that writing into a pipe should be OK, since pipes are
> non-seekable.  However, with pipes in this situation you still have a
> problem if processes try to write more than PIPE_BUF bytes.

You aren't using a pipe there.  What you are doing is having the shell 
open the file, then the md5sum processes all inherit that fd so they all 
share the same offset.  As long as they write() the entire line at once, 
the file pointer will be updated atomically for all processes and the 
lines from each process won't clobber each other.