[GNUnet-SVN] r2713 - in doodle: . doc/man src/doodle

[grothoff]

Author: grothoff
Date: 2006-05-02 14:30:51 -0700 (Tue, 02 May 2006)
New Revision: 2713

Modified:
   doodle/ChangeLog
   doodle/doc/man/doodled.1
   doodle/src/doodle/doodled.c
   doodle/src/doodle/shutdown.c
Log:
fixing Mantis 1017

Modified: doodle/ChangeLog
===================================================================
--- doodle/ChangeLog    2006-05-02 20:19:26 UTC (rev 2712)
+++ doodle/ChangeLog    2006-05-02 21:30:51 UTC (rev 2713)
@@ -1,3 +1,14 @@
+Tue May  2 14:56:42 PDT 2006
+       doodled shutdown sequence was broken, fixed.
+       doodled used stat instead of lstat, resulting in
+       symbolic links being added and then removed from
+       the database; fixed.  doodled no longer verifies
+       all entries in the database on startup (too costly).
+       We assume that "doodle -b" has been used to create
+       an up-to-date database and that doodled will always
+       run in the background keeping it (sufficiently)
+       up-to-date.
+
 Sat Jan  7 02:37:53 PST 2006
        Added Swedish translation.
 

Modified: doodle/doc/man/doodled.1
===================================================================
--- doodle/doc/man/doodled.1    2006-05-02 20:19:26 UTC (rev 2712)
+++ doodle/doc/man/doodled.1    2006-05-02 21:30:51 UTC (rev 2713)
@@ -1,7 +1,7 @@
-.TH DOODLED "1" "Dec 26 2004" "doodled"
+.TH DOODLED "1" "May 2 2006" "doodled"
 
 .SH "NAME"
-doodled \- a daemon to keep your doodle database up-to-date
+doodled \- a daemon to keep your doodle database up\-to\-date
 
 .SH "SYNOPSIS"
 .B doodled
@@ -9,12 +9,12 @@
 
 .SH "DESCRIPTION"
 .PP
-doodled monitors your filesystem for changes and updates the doodle database 
to keep it always up-to-date with the latest data.  doodled is highly 
experimental at this point.  It uses libfam and is thus limited to monitoring 
less than 1024 directories for changes.  Also, there is currently no code to 
ensure that doodle does not read a database that is currently modified by 
doodled.  doodled has barely been tested and thus there is a high chance of 
unexpected bugs.  Treat doodled as alpha-quality software for now, please!
+doodled monitors your filesystem for changes and updates the doodle database 
to keep it always up\-to\-date with the latest data.  doodled is experimental 
at this point.  It uses libfam and is thus limited to monitoring less than 1024 
directories for changes.
 
 .SH "OPTIONS"
 .TP
 .B \-B \fILANG, \fB\-\-binary=\fILANG\fR
-Use the generic plaintext extractor for the language with the 2-letter 
language code LANG.  Supported languages are DA (Danish), DE (German), EN 
(English), ES (Spanish), IT (Italian) and NO (Norwegian).  Use this option to 
enable fulltext indexing (for a particular language).
+Use the generic plaintext extractor for the language with the 2-letter 
language code LANG.  See extract(1) for a list of supported languages.  Use 
this option to enable fulltext indexing (for a particular language).
 .TP 
 \fB\-d \fIFILENAME\fR, \fB\-\-database=\fIFILENAME\fR
 use FILENAME for the location of the database (use when building or 
searching).  This option is particularly useful when doodle is used to search 
different types of files (or is operated with different extractor options).  
Using this option doodle can be used to build specialized indices (i.e. one per 
file system), which can in turn improve search performance.  When searching, 
you can pass a colon-separated list of database file names, in that case all 
databases are searched.  Note that the disk-space consumption of a single 
database is typically slightly smaller than if the database is split into 
multiple files.  Nevertheless, the space\-savings are likely to be small (a few 
percent).  You can also use  the environment variable DOODLE_PATH to set the 
list of database files to search.  The option overrides the environment 
variable if both are used.  If the option is not given and DOODLE_PATH is not 
set, "~/.doodle" is used.
@@ -29,7 +29,7 @@
 print help page
 .TP
 .B \-H \fIALGORITHM\fR, \fB\-\-hash=\fIALGORITHM\fR
-Use the ALGORITHM to compute a hash of each file (possible algorithms are sha1 
and md5).
+Use the ALGORITHM to compute a hash of each file (possible algorithms are md5, 
rmd160 and sha1).
 .TP
 \fB\-l \fILIBRARIES\fR, \fB\-\-library=\fILIBRARIES\fR
 specify which libextractor plugins to use (for building the index with \-b or 
for printing information about files with \-e)
@@ -38,7 +38,7 @@
 log messages to the given logfile.
 .TP
 \fB\-m \fILIMIT\fR\fR, \fB\-\-memory=\fILIMIT\fR
-use at most LIMIT MB of memory for the nodes of the suffix-tree (after that, 
serialize to disk).  Note that a smaller value will reduce memory consumption 
but increase the size of the temporary file (and slow down indexing).  The 
default is 8 MB.
+use at most LIMIT MB of memory for the nodes of the suffix\-tree (after that, 
serialize to disk).  Note that a smaller value will reduce memory consumption 
but increase the size of the temporary file (and slow down indexing).  The 
default is 8 MB.
 .TP
 \fB\-n\fR, \fB\-\-nodefault\fR
 do not load the default set of plugins (only load plugins specified with \-l)
@@ -55,17 +55,17 @@
 .SH "ENVIRONMENT"
 .TP
 .B DOODLE_PATH
-Colon-separated list of databases to search.  Note that when building the 
database this path must either only contain one filename or the option 
\fB\-b\fP must be used to specify the database file.  Default is "~/.doodle".
+Colon\-separated list of databases to search.  Note that when building the 
database this path must either only contain one filename or the option 
\fB\-b\fP must be used to specify the database file.  Default is "~/.doodle".
 .TP
 .B PRUNEPATHS
-Space-separated list of paths to exclude.  Can be overridden with the 
\fB\-P\fR option.  
+Space\-separated list of paths to exclude.  Can be overridden with the 
\fB\-P\fR option.  
 
 .SH "NOTES"
 Doodle depends on libextractor.  You can download libextractor
 from http://gnunet.org/libextractor/.
 
 .SH "SEE ALSO"
-\fBdoodle\fP(1), \fBfam\fP(3), \fBlibdoodle\fP(3)
+\fBextract\fP(1), \fBdoodle\fP(1), \fBfam\fP(3), \fBlibdoodle\fP(3)
 
 .SH "LEGAL NOTICE"
 libdoodle and doodle are released under the GPL.
@@ -74,7 +74,7 @@
 Report bugs to mantis <http://gnunet.org/mantis/> or by sending electronic 
mail to <address@hidden>
 
 .SH "AUTHORS"
-doodle was originally written by Christian Grothoff <address@hidden>.
+doodle and doodled were originally written by Christian Grothoff 
<address@hidden>.
 
 .SH "AVAILABILITY"
 You can obtain the original author's latest version from 
http://gnunet.org/doodle/.

Modified: doodle/src/doodle/doodled.c
===================================================================
--- doodle/src/doodle/doodled.c 2006-05-02 20:19:26 UTC (rev 2712)
+++ doodle/src/doodle/doodled.c 2006-05-02 21:30:51 UTC (rev 2713)
@@ -1,6 +1,6 @@
 /*
      This file is part of doodle.
-     (C) 2004 Christian Grothoff (and other contributing authors)
+     (C) 2004, 2006 Christian Grothoff (and other contributing authors)
 
      doodle is free software; you can redistribute it and/or modify
      it under the terms of the GNU General Public License as published
@@ -39,6 +39,7 @@
 #include <netinet/in.h>
 #include <stdarg.h>
 #include <dirent.h>
+#include <signal.h>
 #include <fam.h>
 #include <locale.h>
 #include "doodle.h"
@@ -175,7 +176,8 @@
  * to process FAM events.  Puts all events into the
  * queue for the main worker.
  */
-static void * processEvents(DIC * cls) {
+static void * processEvents(void * arg) {
+  DIC * cls = arg;
   FAMEvent fe;
   fd_set r;
   fd_set w;
@@ -323,8 +325,8 @@
     }
   }
 
-  if (0 != stat(filename,
-               &sbuf)) {
+  if (0 != lstat(filename,
+                &sbuf)) {
     dic->log(dic->logContext,
             DOODLE_LOG_VERY_VERBOSE,
             _("Call to '%s' for file '%s' failed: %s\n"),
@@ -381,11 +383,6 @@
             _("Will monitor directory '%s' for changes.\n"),
             filename);
     fn = STRDUP(filename);
-
-    /* make sure FAM queue is empty, otherwise the
-       next call may block forever; we might want
-       to do this with threading, but for that
-       we would need to start pthreads... */
     if (0 == FAMMonitorDirectory(&dic->fc,
                                 filename,
                                 &dic->fr[dic->frPos],
@@ -446,7 +443,8 @@
 /**
  * Main worker thread.  Register FAM events and process.
  */
-static void * worker(DIC * cls) {
+static void * worker(void * arg) {
+  DIC * cls = arg;
   int i;
   int more;
   int wasMore;
@@ -463,7 +461,7 @@
   cls->events = NULL;
   cls->signal = SEMAPHORE_NEW(0);
   if (0 != PTHREAD_CREATE(&helperThread,
-                         (PThreadMain) &processEvents,
+                         &processEvents,
                          cls,
                          64 * 1024)) {
     cls->log(cls->logContext,
@@ -473,9 +471,6 @@
     return NULL;
   }
 
-  DOODLE_tree_truncate_modified(cls->tree,
-                               cls->log,
-                               cls->logContext);
   cls->log(cls->logContext,
           DOODLE_LOG_VERBOSE,
           _("Registering with FAM for file system events.\n"));
@@ -548,6 +543,12 @@
   } /* forever (until signal) */
 
   cls->continueRunning = 0;
+  if (0 != FAMClose(&cls->fc)) {
+    cls->log(cls->logContext,
+          DOODLE_LOG_CRITICAL,
+          _("Error disconnecting from fam.\n"));
+  }
+  PTHREAD_KILL(&helperThread, SIGTERM);
   PTHREAD_JOIN(&helperThread, &unused);
   SEMAPHORE_FREE(cls->signal);
 
@@ -574,7 +575,6 @@
   FILE * logfile;
   PTHREAD_T workerThread;
   void * unused;
-  FAMEvent fe;
 
   cls.argc = argc;
   cls.argv = argv;
@@ -668,7 +668,7 @@
 
   MUTEX_CREATE(&cls.lock);
   if (0 != PTHREAD_CREATE(&workerThread,
-                         (PThreadMain)&worker,
+                         &worker,
                          &cls,
                          64 * 1024)) {
     my_log(logfile,
@@ -678,6 +678,8 @@
     ret = -1;
   } else {
     wait_for_shutdown();
+    cls.continueRunning = 0;
+    SEMAPHORE_UP(cls.signal);
     PTHREAD_JOIN(&workerThread, &unused);
   }
   MUTEX_DESTROY(&cls.lock);
@@ -693,30 +695,11 @@
 
 
   for (i=0;i<cls.frSize;i++) {
-    /* flush fam-event queue */
-    while (FAMPending(&cls.fc))
-      if (-1 == FAMNextEvent(&cls.fc, &fe)) {
-       cls.log(cls.logContext,
-               DOODLE_LOG_CRITICAL,
-               _("Call to '%s' failed: %s\n"),
-               "FAMNextEvent",
-               FamErrlist[FAMErrno]);
-       break; /* avoid persistent error */
-      }
     if (cls.frNames[i] != NULL) {
       my_log(logfile,
             DOODLE_LOG_VERBOSE,
             _("Cancelling fam monitor '%s'.\n"),
             cls.frNames[i]);
-      if (-1 == FAMCancelMonitor(&cls.fc,
-                                &cls.fr[i])) {
-       cls.log(cls.logContext,
-               DOODLE_LOG_CRITICAL,
-               _("Call to '%s' for file '%s' failed: %s\n"),
-               "FAMCancelMonitor",
-               cls.frNames[i],
-               FamErrlist[FAMErrno]);
-      }
       free(cls.frNames[i]);
     }
   }
@@ -734,11 +717,6 @@
   GROW(cls.frNames,
        cls.frSize,
        0);
-  if (0 != FAMClose(&cls.fc)) {
-    my_log(logfile,
-          DOODLE_LOG_CRITICAL,
-          _("Error disconnecting from fam.\n"));
-  }
   my_log(logfile,
         DOODLE_LOG_VERBOSE,
         _("Unloading libextractor plugins.\n"));

Modified: doodle/src/doodle/shutdown.c
===================================================================
--- doodle/src/doodle/shutdown.c        2006-05-02 20:19:26 UTC (rev 2712)
+++ doodle/src/doodle/shutdown.c        2006-05-02 21:30:51 UTC (rev 2713)
@@ -44,7 +44,6 @@
  * @param signum is ignored
  */
 void run_shutdown(int signum) {
-  /*  printf("Shutdown handler %d\n", signum); */
   if (shutdown_signal != NULL) {
     shutdown_active = 1;
     SEMAPHORE_UP(shutdown_signal);