bug#8490: dd reads random number of records from pipes - named or otherwise - coreutils 8.9

[Eric Blake]

On 04/12/2011 03:02 PM, Jesse Gordon wrote:
> I can't believe such an obvious bug would exist this long, but on the
> other hand the test is so simple I can't see where it's user error.

Thanks for the report.  And you are correct in surmising that it is user
error and not a bug in dd.

> dd, when reading from stdin or from a named pipe sometimes (but not
> always) reads a random number of records a bit less then what it should.

Rather, dd reads as many bytes as possible, but unless that is less than
PIPE_MAX, it is not guaranteed to be an atomic read.  In turn, if you
have asked dd to pad out partial reads into complete writes, then that
explains your problem.  Unfortunately, it is rather easy to do this
without realizing it; the POSIX wording on how dd behaves is rather
detailed.

> I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even 
> mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed.
> However, if=arealfile seems to always work perfectly.
> 
> To replicate:
> 
> yes|dd bs=1000 count=1000|wc -c

There's your problem.  bs=1000 is the key that tells dd to always write
1000 byte output blocks, even if the input block hit a short read, and
stop after 1000 reads.

Instead, try:

yes|dd ibs=1000 obs=1000 count=1000|wc -c

which tells dd to explicitly read in input blocks of 1000 bytes, even if
it requires multiple reads, prior to doing output blocks of 1000 bytes,
and stop after 1000 writes.

> 
> The problem exists on the following coronations:

It exists everywhere that dd complies with POSIX, even with non-GNU dd.
 Because POSIX requires the difference in behavior between bs=nnn vs.
ibs=nnn obs=nnn.

-- 
Eric Blake   address@hidden    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

signature.asc
Description: OpenPGP digital signature