Re: dd patch to output transfer rate, byte count, and time

[Paul Eggert]

Paul Eggert <address@hidden> writes:

> Come to think of it, if we wanted to support progress-bar add-ons, the
> output line should also contain a full byte count,

While adding this I also tuned the diagnostic a bit (using more-standard
SI forms, e.g.).  And I discovered that we do need to be able to turn off
the new diagnostic in order to do regression tests, since the diagnostic
contains timing info.

I installed the following patch, which introduces (for lack of a
better name) a status=noxfer operand to dd.

2004-11-15  Paul Eggert  <address@hidden>

        * NEWS: New dd operand "status=noxfer".
        * src/dd.c (C_ASCII, C_EBCDIC, C_IBM, C_BLOCK, C_UNBLOCK,
        C_LCASE, C_UCASE, C_SWAB, C_NOERROR, C_NOTRUNC, C_SYNC, C_TWOBUFS,
        C_NOCREAT, C_EXCL, C_FDATASYNC, C_FSYNC): Now constants, not
        macros.
        (STATUS_NOXFER, statuses): New constants.
        (usage, print_stats, scanargs): Add support for status=noxfer.
        (usage): Update status output to match new behavior.
        (print_stats): Always output complete byte count.
        Put space between numbers and units, as SI requires.
        Use ngettext so that i18n can use plurals for "byte" and "second".
        Don't multiply by 1e-9 (inexact); divide by 1e9 (which is exact).
        (iflag_error_msgid, oflag_error_msgid): Remove; replace uses by
        the string.
        * tests/dd/skip-seek (@Tests): Use status=noxfer to avoid
        problems with regression testing.

        * doc/coreutils.texi (dd invocation): Reword the new dd message.

        * lib/human.h (LONGEST_HUMAN_READABLE): Add 1 for space before unit.
        (human_space_before_unit): New constant.
        * lib/human.c (human_readable): Support it.

Index: NEWS
===================================================================
RCS file: /fetish/cu/NEWS,v
retrieving revision 1.249
diff -p -u -r1.249 NEWS
--- NEWS        15 Nov 2004 06:49:08 -0000      1.249
+++ NEWS        16 Nov 2004 04:48:24 -0000
@@ -182,8 +182,8 @@ GNU coreutils NEWS                      
   copying or moving multiple times to the same destination in a file
   system with a coarse time stamp resolution.
 
-  dd now also prints the number of bytes copied, the time, and the
-  transfer rate.
+  dd now also prints the number of bytes transferred, the time, and the
+  transfer rate.  The new "status=noxfer" operand suppresses this change.
 
   dd has new conversions for the conv= option:
 
Index: doc/coreutils.texi
===================================================================
RCS file: /fetish/cu/doc/coreutils.texi,v
retrieving revision 1.225
diff -p -u -r1.225 coreutils.texi
--- doc/coreutils.texi  15 Nov 2004 06:50:23 -0000      1.225
+++ doc/coreutils.texi  16 Nov 2004 04:48:26 -0000
@@ -6872,12 +6872,12 @@ and when @command{dd} completes, it outp
 @example
 $ dd if=/dev/zero of=/dev/null count=10MB & pid=$!
 $ kill -s USR1 $pid; wait $pid
-4111640+0 records in
-4111639+0 records out
-2.1GB copied in 7.95411s (265MB/s)
+3385223+0 records in
+3385223+0 records out
+1733234176 bytes (1.7 GB) copied, 6.42173 seconds, 270 MB/s
 10000000+0 records in
 10000000+0 records out
-5.1GB copied in 19.3794s (264MB/s)
+5120000000 bytes (5.1 GB) copied, 18.913 seconds, 271 MB/s
 @end example
 
 @exitstatus
Index: lib/human.c
===================================================================
RCS file: /fetish/cu/lib/human.c,v
retrieving revision 1.27
diff -p -u -r1.27 human.c
--- lib/human.c 2 Aug 2004 22:58:22 -0000       1.27
+++ lib/human.c 16 Nov 2004 04:48:27 -0000
@@ -158,6 +158,9 @@ group_number (char *number, size_t numbe
    so on.  Numbers smaller than the power aren't modified.
    human_autoscale is normally used together with human_SI.
 
+   If (OPTS & human_space_before_unit), use a space to separate the
+   number from any suffix that is appended as described below.
+
    If (OPTS & human_SI), append an SI prefix indicating which power is
    being used.  If in addition (OPTS & human_B), append "B" (if base
    1000) or "iB" (if base 1024) to the SI prefix.  When ((OPTS &
@@ -384,6 +387,9 @@ human_readable (uintmax_t n, char *buf, 
              break;
        }
 
+      if ((exponent | (opts & human_B)) && (opts & human_space_before_unit))
+       *psuffix++ = ' ';
+
       if (exponent)
        *psuffix++ = (! (opts & human_base_1024) && exponent == 1
                      ? 'k'
Index: lib/human.h
===================================================================
RCS file: /fetish/cu/lib/human.h,v
retrieving revision 1.9
diff -p -u -r1.9 human.h
--- lib/human.h 2 Aug 2004 22:49:58 -0000       1.9
+++ lib/human.h 16 Nov 2004 04:48:27 -0000
@@ -42,10 +42,11 @@
    302 / 1000 is ceil (log10 (2.0)).  Add 1 for integer division truncation.
    Also, the output can have a thousands separator between every digit,
    so multiply by MB_LEN_MAX + 1 and then subtract MB_LEN_MAX.
+   Append 1 for a space before the suffix.
    Finally, append 3, the maximum length of a suffix.  */
 # define LONGEST_HUMAN_READABLE \
   ((2 * sizeof (uintmax_t) * CHAR_BIT * 302 / 1000 + 1) * (MB_LEN_MAX + 1) \
-   - MB_LEN_MAX + 3)
+   - MB_LEN_MAX + 1 + 3)
 
 /* Options for human_readable.  */
 enum
@@ -74,11 +75,14 @@ enum
   /* Prefer base 1024 to base 1000.  */
   human_base_1024 = 32,
 
+  /* Prepend " " before unit symbol.  */
+  human_space_before_unit = 64,
+
   /* Append SI prefix, e.g. "k" or "M".  */
-  human_SI = 64,
+  human_SI = 128,
 
   /* Append "B" (if base 1000) or "iB" (if base 1024) to SI prefix.  */
-  human_B = 128
+  human_B = 256
 };
 
 char *human_readable (uintmax_t, char *, int, uintmax_t, uintmax_t);
Index: src/dd.c
===================================================================
RCS file: /fetish/cu/src/dd.c,v
retrieving revision 1.168
diff -p -u -r1.168 dd.c
--- src/dd.c    15 Nov 2004 06:49:59 -0000      1.168
+++ src/dd.c    16 Nov 2004 04:48:27 -0000
@@ -74,23 +74,36 @@
 #define MAX_BLOCKSIZE MIN (SIZE_MAX, MIN (SSIZE_MAX, OFF_T_MAX))
 
 /* Conversions bit masks. */
-#define C_ASCII 01
-#define C_EBCDIC 02
-#define C_IBM 04
-#define C_BLOCK 010
-#define C_UNBLOCK 020
-#define C_LCASE 040
-#define C_UCASE 0100
-#define C_SWAB 0200
-#define C_NOERROR 0400
-#define C_NOTRUNC 01000
-#define C_SYNC 02000
-/* Use separate input and output buffers, and combine partial input blocks. */
-#define C_TWOBUFS 04000
-#define C_NOCREAT 010000
-#define C_EXCL 020000
-#define C_FDATASYNC 040000
-#define C_FSYNC 0100000
+enum
+  {
+    C_ASCII = 01,
+
+    C_EBCDIC = 02,
+    C_IBM = 04,
+    C_BLOCK = 010,
+    C_UNBLOCK = 020,
+    C_LCASE = 040,
+    C_UCASE = 0100,
+    C_SWAB = 0200,
+    C_NOERROR = 0400,
+    C_NOTRUNC = 01000,
+    C_SYNC = 02000,
+
+    /* Use separate input and output buffers, and combine partial
+       input blocks. */
+    C_TWOBUFS = 04000,
+
+    C_NOCREAT = 010000,
+    C_EXCL = 020000,
+    C_FDATASYNC = 040000,
+    C_FSYNC = 0100000
+  };
+
+/* Status bit masks.  */
+enum
+  {
+    STATUS_NOXFER = 01
+  };
 
 /* The name this program was run with. */
 char *program_name;
@@ -126,6 +139,9 @@ static int conversions_mask = 0;
 static int input_flags = 0;
 static int output_flags = 0;
 
+/* Status flags for what is printed to stderr.  */
+static int status_flags = 0;
+
 /* If nonzero, filter characters through the translation table.  */
 static bool translation_needed = false;
 
@@ -220,6 +236,13 @@ static struct symbol_value const flags[]
   {"",         0}
 };
 
+/* Status, for status="...".  */
+static struct symbol_value const statuses[] =
+{
+  {"noxfer",   STATUS_NOXFER},
+  {"",         0}
+};
+
 /* Translation table formed by applying successive transformations. */
 static unsigned char trans_table[256];
 
@@ -361,6 +384,7 @@ Copy a file, converting and formatting a
   oflag=FLAGS     write as per the comma separated symbol list\n\
   seek=BLOCKS     skip BLOCKS obs-sized blocks at start of output\n\
   skip=BLOCKS     skip BLOCKS ibs-sized blocks at start of input\n\
+  status=noxfer   suppress transfer statistics\n\
 "), stdout);
       fputs (_("\
 \n\
@@ -414,9 +438,9 @@ print I/O statistics to standard error, 
 \n\
   $ dd if=/dev/zero of=/dev/null& pid=$!\n\
   $ kill -USR1 $pid; sleep 1; kill $pid\n\
-  10807656+0 records in\n\
-  10807656+0 records out\n\
-  5.5GB copied in 20.8225s (266MB/s)\n\
+  18335302+0 records in\n\
+  18335302+0 records out\n\
+  9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s\n\
 \n\
 Options are:\n\
 \n\
@@ -452,9 +476,11 @@ print_stats (void)
   char buf[2][MAX (INT_BUFSIZE_BOUND (uintmax_t), LONGEST_HUMAN_READABLE + 1)];
   struct timespec now;
   int human_opts =
-    human_autoscale | human_round_to_nearest | human_SI | human_B;
+    (human_autoscale | human_round_to_nearest
+     | human_space_before_unit | human_SI | human_B);
   uintmax_t start_sec = start_time.tv_sec;
   enum { BILLION = 1000000000 };
+  double BILLIONe0 = BILLION;
   double delta_s;
   char const *bytes_per_second;
 
@@ -472,10 +498,19 @@ print_stats (void)
                : _("truncated records")));
     }
 
+  if (status_flags & STATUS_NOXFER)
+    return;
+
   /* Use integer arithmetic to compute the transfer rate if possible,
      since that makes it easy to use SI abbreviations; otherwise, fall
      back on floating-point without abbreviations.  */
 
+  fprintf (stderr,
+          ngettext ("%s byte (%s) copied",
+                    "%s bytes (%s) copied", w_bytes == 1),
+          umaxtostr (w_bytes, buf[0]),
+          human_readable (w_bytes, buf[1], human_opts, 1, 1));
+
   if ((start_time.tv_sec < now.tv_sec
        || (start_time.tv_sec == now.tv_sec
           && start_time.tv_nsec < now.tv_nsec))
@@ -483,7 +518,7 @@ print_stats (void)
     {
       uintmax_t delta_ns = (BILLION * (now.tv_sec - start_sec)
                            + now.tv_nsec - start_time.tv_nsec);
-      delta_s = delta_ns / 1e9;
+      delta_s = delta_ns / BILLIONe0;
       bytes_per_second = human_readable (w_bytes, buf[1], human_opts,
                                         BILLION, delta_ns);
     }
@@ -491,7 +526,7 @@ print_stats (void)
     {
       delta_s = now.tv_sec;
       delta_s -= start_time.tv_sec;
-      delta_s += 1e-9 * (now.tv_nsec - start_time.tv_nsec);
+      delta_s += (now.tv_nsec - start_time.tv_nsec) / BILLIONe0;
       if (0 < delta_s)
        sprintf (buf[1], "%gB", w_bytes / delta_s);
       else
@@ -499,8 +534,9 @@ print_stats (void)
       bytes_per_second = buf[1];
     }
 
-  fprintf (stderr, _("%s copied in %gs (%s/s)\n"),
-          human_readable (w_bytes, buf[0], human_opts, 1, 1),
+  fprintf (stderr,
+          ngettext (", %g second, %s/s\n",
+                    ", %g seconds, %s/s\n", delta_s == 1),
           delta_s, bytes_per_second);
 }
 
@@ -608,10 +644,6 @@ write_output (void)
   oc = 0;
 }
 
-/* Diagnostics for invalid iflag="..." and oflag="..." symbols.  */
-static char const iflag_error_msgid[] = N_("invalid input flag: %s");
-static char const oflag_error_msgid[] = N_("invalid output flag: %s");
-
 /* Interpret one "conv=..." or similar operand STR according to the
    symbols in TABLE, returning the flags specified.  If the operand
    cannot be parsed, use ERROR_MSGID to generate a diagnostic.
@@ -709,9 +741,14 @@ scanargs (int argc, char **argv)
        conversions_mask |= parse_symbols (val, conversions,
                                           N_("invalid conversion: %s"));
       else if (STREQ (name, "iflag"))
-       input_flags |= parse_symbols (val, flags, iflag_error_msgid);
+       input_flags |= parse_symbols (val, flags,
+                                     N_("invalid input flag: %s"));
       else if (STREQ (name, "oflag"))
-       output_flags |= parse_symbols (val, flags, oflag_error_msgid);
+       output_flags |= parse_symbols (val, flags,
+                                      N_("invalid output flag: %s"));
+      else if (STREQ (name, "status"))
+       status_flags |= parse_symbols (val, statuses,
+                                      N_("invalid status flag: %s"));
       else
        {
          bool invalid = false;
Index: tests/dd/skip-seek
===================================================================
RCS file: /fetish/cu/tests/dd/skip-seek,v
retrieving revision 1.10
diff -p -u -r1.10 skip-seek
--- tests/dd/skip-seek  21 Mar 2004 18:50:30 -0000      1.10
+++ tests/dd/skip-seek  16 Nov 2004 04:48:28 -0000
@@ -27,7 +27,8 @@ my $script_name = $ENV{SCRIPT_NAME};
 my @Tests =
     (
      [
-      'skip-seek-1', qw (bs=1 skip=1 seek=2 conv=notrunc count=3 
address@hidden@ < ),
+      'skip-seek-1',
+      qw (bs=1 skip=1 seek=2 conv=notrunc count=3 status=noxfer 
address@hidden@ < ),
       {IN=> '0123456789abcdef'},
       {AUX=> 'zyxwvutsrqponmlkji'},
       {OUT=> ''},
@@ -35,7 +36,8 @@ my @Tests =
       {CMP=> ['zy123utsrqponmlkji', {'@AUX@'=> undef}]},
      ],
      [
-      'skip-seek-2', qw (bs=5 skip=1 seek=1 conv=notrunc count=1 
address@hidden@ < ),
+      'skip-seek-2',
+      qw (bs=5 skip=1 seek=1 conv=notrunc count=1 status=noxfer 
address@hidden@ < ),
       {IN=> '0123456789abcdef'},
       {AUX=> 'zyxwvutsrqponmlkji'},
       {OUT=> ''},
@@ -43,7 +45,8 @@ my @Tests =
       {CMP=> ['zyxwv56789ponmlkji', {'@AUX@'=> undef}]},
      ],
      [
-      'skip-seek-3', qw (bs=5 skip=1 seek=1 count=1 address@hidden@ < ),
+      'skip-seek-3',
+      qw (bs=5 skip=1 seek=1 count=1 status=noxfer address@hidden@ < ),
       {IN=> '0123456789abcdef'},
       {AUX=> 'zyxwvutsrqponmlkji'},
       {OUT=> ''},
@@ -53,7 +56,7 @@ my @Tests =
      [
       # Before fileutils-4.0.45, the last 10 bytes of output
       # were these "\0\0\0\0\0\0\0\0  ".
-      'block-sync-1', qw(ibs=10 cbs=10), 'conv=block,sync', '<',
+      'block-sync-1', qw(ibs=10 cbs=10 status=noxfer), 'conv=block,sync', '<',
       {IN=> "01234567\nabcdefghijkl\n"},
       {OUT=> "01234567  abcdefghij          "},
       {ERR=> "2+1 records in\n0+1 records out\n1 truncated record\n"},