[bug-idutils] [PATCH 1/4] mkid: use ftello (not ftell) and fail if an offset is 2^32 or larger

[Jim Meyering]

FYI,

>From eb170e813979f81b9103f51d3616de29389e4513 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Wed, 20 Jan 2010 21:31:40 +0100
Subject: [PATCH 1/4] mkid: use ftello (not ftell) and fail if an offset is 2^32 
or larger

This is necessary because the internal layout requires that an
offset be representable as a 4-byte quantity.
* src/mkid.c (write_id_file): Use ftello, not ftell.
The latter would fail on files larger than 4GiB.  Now,
we still fail for such files, but use ftello instead --
and give a diagnostic.
---
 gnulib     |    2 +-
 src/mkid.c |   15 ++++++++++++---
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/gnulib b/gnulib
index 0c6cf5a..880f2b6 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit 0c6cf5ab43555377b99d94febb2d6f23fc3d2cb0
+Subproject commit 880f2b69df57af506439d6aaf1fe185a6f960e43
diff --git a/src/mkid.c b/src/mkid.c
index 59ac41a..b1a0fa9 100644
--- a/src/mkid.c
+++ b/src/mkid.c
@@ -710,14 +710,20 @@ write_id_file (struct idhead *idhp)
   /* write out the list of pathnames */

   fseek (idhp->idh_FILE, sizeof_idhead (), 0);
-  idhp->idh_flinks_offset = ftell (idhp->idh_FILE);
+  off_t off = ftello (idhp->idh_FILE);
+  if (UINT32_MAX < off)
+    error (EXIT_FAILURE, 0, _("internal limitation: offset of 2^32 or 
larger"));
+  idhp->idh_flinks_offset = off;
   serialize_file_links (idhp);

   /* write out the list of identifiers */

   putc ('\0', idhp->idh_FILE);
   putc ('\0', idhp->idh_FILE);
-  idhp->idh_tokens_offset = ftell (idhp->idh_FILE);
+  off = ftello (idhp->idh_FILE);
+  if (UINT32_MAX < off)
+    error (EXIT_FAILURE, 0, _("internal limitation: offset of 2^32 or 
larger"));
+  idhp->idh_tokens_offset = off;

   for (i = 0; i < token_table.ht_fill; i++, tokens++)
     {
@@ -761,7 +767,10 @@ write_id_file (struct idhead *idhp)
     }
   assert_hits (summary_root);
   idhp->idh_tokens = token_table.ht_fill;
-  output_length = ftell (idhp->idh_FILE);
+  off = ftello (idhp->idh_FILE);
+  if (UINT32_MAX < off)
+    error (EXIT_FAILURE, 0, _("internal limitation: offset of 2^32 or 
larger"));
+  output_length = off;
   idhp->idh_end_offset = output_length - 2;
   idhp->idh_buf_size = max_buf_size;
   idhp->idh_vec_size = max_vec_size;
--
1.7.1.189.g07419


>From 4900e6847bf8ef3be13efb48ed1008addf9dfa32 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Tue, 30 Mar 2010 13:36:26 +0200
Subject: [PATCH 2/4] doc: add @dircategory to Info documentation

The Texinfo documentation says that Info files should use @dircategory,
and suggests consulting the Free Software Directory to select a category
name.  The directory places under ""Text creation and manipulation".
* doc/idutils.texi: Add "@dircategory Text creation and manipulation".
Inspired by a nearly identical patch by Colin Watson for Parted.
---
 doc/idutils.texi |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/doc/idutils.texi b/doc/idutils.texi
index a047adf..2b68233 100644
--- a/doc/idutils.texi
+++ b/doc/idutils.texi
@@ -18,13 +18,15 @@
 @syncodeindex pg cp
 @syncodeindex vr cp

address@hidden Text creation and manipulation
 @ifinfo
 @format
 START-INFO-DIR-ENTRY
-* ID database: (idutils).              Identifier database utilities.
+* ID utilities: (idutils).             Indexing and searching utilities.
 * mkid: (idutils)mkid invocation.      Creating an ID database.
 * lid: (idutils)lid invocation.        Matching words and patterns.
 * fid: (idutils)fid invocation.        Listing a file's tokens.
+* gid: (idutils)gid invocation.        Token-based grep.
 * fnid: (idutils)fnid invocation.      Looking up file names.
 * xtokid: (idutils)xtokid invocation.  Testing mkid scanners.
 END-INFO-DIR-ENTRY
@@ -78,6 +80,7 @@ This manual documents version @value{VERSION} of the ID 
utilities.
 * mkid invocation::             Creating an ID database.
 * lid invocation::              Querying an ID database by token.
 * fid invocation::              Listing a file's tokens.
+* gid invocation::              Token-based grep
 * fnid invocation::             Looking up file names.
 * xtokid invocation::           Testing language scanners.
 * Past and Future::             History and future directions.
--
1.7.1.189.g07419


>From d958af18d07bd4c01685f101a75c691c35a30089 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Tue, 30 Mar 2010 13:43:44 +0200
Subject: [PATCH 3/4] doc: add "gid" to the menus and direntry list

* doc/idutils.texi (gid invocation): Add a section on "gid".
---
 doc/idutils.texi |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/doc/idutils.texi b/doc/idutils.texi
index 2b68233..29fd44b 100644
--- a/doc/idutils.texi
+++ b/doc/idutils.texi
@@ -1259,6 +1259,25 @@ fnid \*.c
 here protects the @samp{*} from being expanded by the shell.)

 @c ************* gkm *********************************************************
address@hidden gid invocation
address@hidden @code{gid}: Token-based grep
+
address@hidden gid
address@hidden searching files, grep
+
address@hidden lists each line containing one of the specified tokens.
+
+Running @command{gid} is equivalent to @samp{lid -R grep})
+
+For example, the command:
+
address@hidden
+gid important_variable
address@hidden example
+
address@hidden lists each line of each file in the database that contains that 
token.
+
address@hidden ************* gkm 
*********************************************************
 @node xtokid invocation
 @chapter @file{xtokid}: Testing Language Scanners

--
1.7.1.189.g07419


>From 080106b3d8ae87d8851c489b32175b98e4978c4e Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Sun, 9 May 2010 20:35:48 +0200
Subject: [PATCH 4/4] build: update from gnulib

* bootstrap: Update from gnulib.
* testsuite/init.sh: Update from gnulib.
* gnulib: Update to latest.
---
 bootstrap         |   97 +++++++++++++++++++++++++++++++++++++---------------
 testsuite/init.sh |    9 ++++-
 2 files changed, 76 insertions(+), 30 deletions(-)

diff --git a/bootstrap b/bootstrap
index 456b0b2..a9a778a 100755
--- a/bootstrap
+++ b/bootstrap
@@ -1,4 +1,6 @@
 #! /bin/sh
+# Print a version string.
+scriptversion=2010-04-30.16; # UTC

 # Bootstrap this package from checked-out sources.

@@ -17,7 +19,15 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.

-# Written by Paul Eggert.
+# Originally written by Paul Eggert.  The canonical version of this
+# script is maintained as build-aux/bootstrap in gnulib, however, to
+# be useful to your project, you should place a copy of it under
+# version control in the top-level directory of your project.  The
+# intent is that all customization can be done with a bootstrap.conf
+# file also maintained in your version control; gnulib comes with a
+# template build-aux/bootstrap.conf to get you started.
+
+# Please report bugs or propose patches to address@hidden

 nl='
 '
@@ -43,7 +53,7 @@ Options:
                           sources reside.  Use this if you already
                           have gnulib sources on your machine, and
                           do not want to waste your bandwidth downloading
-                          them again.
+                          them again.  Defaults to \$GNULIB_SRCDIR.
  --copy                   Copy files instead of creating symbolic links.
  --force                  Attempt to bootstrap even if the sources seem
                           not to have been checked out.
@@ -71,6 +81,7 @@ gnulib_modules=
 gnulib_files=

 # A function to be called after everything else in this script.
+# Override it via your own definition in bootstrap.conf.
 bootstrap_epilogue() { :; }

 # The command to download all .po files for a specified domain into
@@ -132,7 +143,8 @@ XGETTEXT_OPTIONS='\\\
  --flag=error:3:c-format --flag=error_at_line:5:c-format\\\
 '

-# Package bug report address for gettext files
+# Package bug report address and copyright holder for gettext files
+COPYRIGHT_HOLDER='Free Software Foundation, Inc.'
 address@hidden

 # Files we don't want to import.
@@ -159,7 +171,6 @@ vc_ignore=auto
 # die otherwise.
 find_tool ()
 {
-  # Find sha1sum, named gsha1sum on MacPorts.
   find_tool_envvar=$1
   shift
   find_tool_names=$@
@@ -187,8 +198,8 @@ find_tool ()
   eval "export $find_tool_envvar"
 }

-# Find sha1sum, named gsha1sum on MacPorts.
-find_tool SHA1SUM sha1sum gsha1sum
+# Find sha1sum, named gsha1sum on MacPorts, and shasum on MacOS 10.6.
+find_tool SHA1SUM sha1sum gsha1sum shasum

 # Override the default configuration, if necessary.
 # Make sure that bootstrap.conf is sourced from the current directory
@@ -313,10 +324,20 @@ get_version() {
   $app --version >/dev/null 2>&1 || return 1

   $app --version 2>&1 |
-  sed -n 's/[^0-9.]*\([0-9]\{1,\}\.[.a-z0-9-]*\).*/\1/p
+  sed -n '# extract version within line
+          s/.*[v ]\{1,\}\([0-9]\{1,\}\.[.a-z0-9-]*\).*/\1/
+          t done
+
+          # extract version at start of line
+          s/^\([0-9]\{1,\}\.[.a-z0-9-]*\).*/\1/
           t done
+
           d
+
           :done
+          #the following essentially does s/5.005/5.5/
+          s/\.0*\([1-9]\)/.\1/g
+          p
           q'
 }

@@ -379,7 +400,7 @@ fi

 cleanup_gnulib() {
   status=$?
-  rm -fr gnulib
+  rm -fr "$gnulib_path"
   exit $status
 }

@@ -387,6 +408,8 @@ git_modules_config () {
   test -f .gitmodules && git config --file .gitmodules "$@"
 }

+gnulib_path=`git_modules_config submodule.gnulib.path`
+
 # Get gnulib files.

 case ${GNULIB_SRCDIR--} in
@@ -396,30 +419,43 @@ case ${GNULIB_SRCDIR--} in
     git submodule init || exit $?
     git submodule update || exit $?

-  elif [ ! -d gnulib ]; then
+  elif [ ! -d "$gnulib_path" ]; then
     echo "$0: getting gnulib files..."

     trap cleanup_gnulib 1 2 13 15

-    git clone --help|grep depth > /dev/null && shallow='--depth 2' || shallow=
-    git clone $shallow git://git.sv.gnu.org/gnulib ||
+    git clone -h|grep -- --depth > /dev/null && shallow='--depth 2' || shallow=
+    git clone $shallow git://git.sv.gnu.org/gnulib "$gnulib_path" ||
       cleanup_gnulib

     trap - 1 2 13 15
   fi
-  GNULIB_SRCDIR=gnulib
+  GNULIB_SRCDIR=$gnulib_path
   ;;
 *)
-  # Redirect the gnulib submodule to the directory on the command line
-  # if possible.
+  # Use GNULIB_SRCDIR as a reference.
   if test -d "$GNULIB_SRCDIR"/.git && \
         git_modules_config submodule.gnulib.url >/dev/null; then
-    git submodule init
-    GNULIB_SRCDIR=`cd $GNULIB_SRCDIR && pwd`
-    git config --replace-all submodule.gnulib.url $GNULIB_SRCDIR
     echo "$0: getting gnulib files..."
-    git submodule update || exit $?
-    GNULIB_SRCDIR=gnulib
+    if git submodule -h|grep -- --reference > /dev/null; then
+      # Prefer the one-liner available in git 1.6.4 or newer.
+      git submodule update --init --reference "$GNULIB_SRCDIR" \
+        "$gnulib_path" || exit $?
+    else
+      # This fallback allows at least git 1.5.5.
+      if test -f "$gnulib_path"/gnulib-tool; then
+        # Since file already exists, assume submodule init already complete.
+        git submodule update || exit $?
+      else
+        # Older git can't clone into an empty directory.
+        rmdir "$gnulib_path" 2>/dev/null
+        git clone --reference "$GNULIB_SRCDIR" \
+          "$(git_modules_config submodule.gnulib.url)" "$gnulib_path" \
+          && git submodule init && git submodule update \
+          || exit $?
+      fi
+    fi
+    GNULIB_SRCDIR=$gnulib_path
   fi
   ;;
 esac
@@ -452,7 +488,7 @@ update_po_files() {
   test -d $ref_po_dir || mkdir $ref_po_dir || return
   download_po_files $ref_po_dir $domain \
     && ls "$ref_po_dir"/*.po 2>/dev/null |
-      sed 's|.*/||; s|\.po$||' > "$po_dir/LINGUAS"
+      sed 's|.*/||; s|\.po$||' > "$po_dir/LINGUAS" || return

   langs=`cd $ref_po_dir && echo *.po|sed 's/\.po//g'`
   test "$langs" = '*' && langs=x
@@ -625,7 +661,7 @@ slurp() {
       for excluded_file in $excluded_files; do
         test "$dir/$file" = "$excluded_file" && continue 2
       done
-      if test $file = Makefile.am; then
+      if test $file = Makefile.am && test "X$gnulib_mk" != XMakefile.am; then
         copied=$copied${sep}$gnulib_mk; sep=$nl
         remove_intl='/^[^#].*\/intl/s/^/#/;'"s!$bt_regex/!!g"
         sed "$remove_intl" $1/$dir/$file | cmp - $dir/$gnulib_mk > /dev/null 
|| {
@@ -643,7 +679,7 @@ slurp() {
           rm -f $dir/$file
           sed '
             /^AC_DEFUN(\[AM_INTL_SUBDIR],/,/^]/c\
-              AC_DEFUN([AM_INTL_SUBDIR], [
+              AC_DEFUN([AM_INTL_SUBDIR], [])
             /^AC_DEFUN(\[gt_INTL_SUBDIR_CORE],/,/^]/c\
               AC_DEFUN([gt_INTL_SUBDIR_CORE], [])
             $a\
@@ -732,7 +768,7 @@ find "$m4_base" "$source_base" \
 # Reconfigure, getting other files.

 # Skip autoheader if it's not needed.
-grep '^[        ]*AC_CONFIG_HEADERS\>' configure.ac >/dev/null ||
+grep -E '^[     ]*AC_CONFIG_HEADERS?\>' configure.ac >/dev/null ||
   AUTOHEADER=true

 for command in \
@@ -776,13 +812,14 @@ if test $with_gettext = yes; then
   rm -f po/Makevars
   sed '
     /^EXTRA_LOCALE_CATEGORIES *=/s/=.*/= '"$EXTRA_LOCALE_CATEGORIES"'/
-    /^MSGID_BUGS_ADDRESS *=/s/=.*/= '"$MSGID_BUGS_ADDRESS"'/
+    /^COPYRIGHT_HOLDER *=/s/=.*/= '"$COPYRIGHT_HOLDER"'/
+    /^MSGID_BUGS_ADDRESS *=/s|=.*|= '"$MSGID_BUGS_ADDRESS"'|
     /^XGETTEXT_OPTIONS *=/{
       s/$/ \\/
       a\
           '"$XGETTEXT_OPTIONS"' $${end_of_xgettext_options+}
     }
-  ' po/Makevars.template >po/Makevars
+  ' po/Makevars.template >po/Makevars || exit 1

   if test -d runtime-po; then
     # Similarly for runtime-po/Makevars, but not quite the same.
@@ -796,7 +833,7 @@ if test $with_gettext = yes; then
         a\
             '"$XGETTEXT_OPTIONS_RUNTIME"' $${end_of_xgettext_options+}
       }
-    ' <po/Makevars.template >runtime-po/Makevars
+    ' po/Makevars.template >runtime-po/Makevars || exit 1

     # Copy identical files from po to runtime-po.
     (cd po && cp -p Makefile.in.in *-quot *.header *.sed *.sin ../runtime-po)
@@ -807,6 +844,10 @@ bootstrap_epilogue

 echo "$0: done.  Now you can run './configure'."

-# Local Variables:
-# indent-tabs-mode: nil
+# Local variables:
+# eval: (add-hook 'write-file-hooks 'time-stamp)
+# time-stamp-start: "scriptversion="
+# time-stamp-format: "%:y-%02m-%02d.%02H"
+# time-stamp-time-zone: "UTC"
+# time-stamp-end: "; # UTC"
 # End:
diff --git a/testsuite/init.sh b/testsuite/init.sh
index 512e876..7772736 100644
--- a/testsuite/init.sh
+++ b/testsuite/init.sh
@@ -21,11 +21,16 @@
 # The typical skeleton of a test looks like this:
 #
 #   #!/bin/sh
-#   : ${srcdir=.}
-#   . "$srcdir/init.sh"; path_prepend_ .
+#   . "${srcdir=.}/init.sh"; path_prepend_ .
 #   Execute some commands.
 #   Note that these commands are executed in a subdirectory, therefore you
 #   need to prepend "../" to relative filenames in the build directory.
+#   Note that the "path_prepend_ ." is useful only if the body of your
+#   test invokes programs residing in the initial directory.
+#   For example, if the programs you want to test are in src/, and this test
+#   script is named tests/test-1, then you would use "path_prepend_ ../src",
+#   or perhaps export PATH='$(abs_top_builddir)/src$(PATH_SEPARATOR)'"$$PATH"
+#   to all tests via automake's TESTS_ENVIRONMENT.
 #   Set the exit code 0 for success, 77 for skipped, or 1 or other for failure.
 #   Use the skip_ and fail_ functions to print a diagnostic and then exit
 #   with the corresponding exit code.
--
1.7.1.189.g07419