Re: make 'dd conv=sync' pad last block

[Scott Weikart]

On Sunday 06 June 2004  3:38 pm, Paul Eggert wrote:
> And some experimentation shows that Solaris 9 'dd' pads input
> blocks, not output blocks:
>
>    $ echo hello | dd obs=1024k of=output conv=sync
>    0+1 records in
>    0+1 records out
>    $ ls -l output
>    -rw-rw-r--   1 eggert   eggert       512 Jun  6 15:28 output

You're right.  [Since their example was writing to a tape, I assumed
they'd "do the right thing", i.e. output fixed-lenth blocks.]

So, I now accept your argument that conv=sync shouldn't pad output
blocks.

>>> You might try "dd ibs=128k obs=128k conv=sync".
>>
>> That technically works, but the results are not desirable.
>>
>> It reads a variable number of 2b block from tar, pads the input
>> buffer to 128KB, then writes it out.
>
> how about this instead?
>
>    tar cbf 1 - whatever | dd obs=1024k | dd bs=1024k conv=sync

That works the same as your first suggestion.

If the obs from your first example has a different value then the
ibs, e.g.

  tar cbf 2 - whatever | dd ibs=2b obs=128k conv=sync

then the output is as small as it should be.  Except the output is
now *too* small: the last block is a partial block.

I'm using the 'st' driver in the linux kernel, and trying to use all
its efficiency options.  The efficiency options for writing
('buffer-writes' and 'async-writes') seem to work with
variable-length blocks.  But, when reading, you need to set the
driver to fixed-length blocks in order to use the 'read-ahead'
option.

So, when you read with 'read-ahead', but the tape has a shorter last
block, you'll get an error on the last block

  kernel: st0: Incorrect block size.

and this final "short" block won't be read, so tar fails with

  tar: Unexpected EOF in archive
  tar: Error is not recoverable: exiting now

> Perhaps GNU 'dd' should have an option to pad output blocks, so that
> you don't need two processes to solve your problem.

I'd be happy to re-submit my patch with a new 'conv' option for the
new behavior.  We could call the new 'conv' option something like

  syncout
  syncoutlast
  synclast

  pad
  padout
  padlast

  integral
  wholeblocks

[I would also include updates to the documentation and ChangeLog in
my patch.]

-scott