Re: [Help-gnunet] severe performance problems

[Christian Grothoff]

Ok, let me summarize:

a) you're using a 200 Mhz CPU.  This is sufficient, but barely above the 100
    Mhz minimum requirements.
b) you're experiencing high CPU utilization, but can't tell which process is
    the cause (gnunetd does not show up in top)
c) you're using an encrypted partition to store the GNUnet data

Well, c) causes the kernel (!) to perform encryption/decryption on any IO 
operation.  That's where your CPU is going, and since the kernel has no PID, 
this is also why it does not show up for any process in b).  Also, c) is a 
bad idea -- GNUnet already (!) encrypts all of the data that is being stored 
in the database, encrypting it again gives you very little additional 
security.  Worse, it seems to me that it kills your CPU (at the IO-rate that 
GNUnet has -- remember, on many machines GNUnet can be IO-bound; on yours it 
is likely CPU bound to start with, and by putting encryption on the disk you 
make it worse).

As for memory consumption:
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   419 gnunet    16   0  136m  70m 5164 S  0.0 45.4   0:07.94 gnunetd
 
The difference between 136 and 70 MB is likely debug symbols and other stuff 
that is not even loaded into memory.  Either way, only RES (70 MB) matters 
for performance (and this is also the reason why your "free" memory does not 
look as bad as you expected it to).  You should be able to reduce RES by 
compiling GNUnet with the right options to reduce code size.  Here is some 
text from doc/memoryuse.txt that maybe of help:


List of the top known memory users in GNUnet for
the default configuration:

* Bloomfilters (size depends on DISKQUOTA, think 64k for 1 MB)
* GAP routing table (100 * TABLESIZE,  think 100k (min) to 7 MB (default) or 
more)
* connection send buffers (variable)
* TCP buffers (variable)
* querymanager (think 64k)
* databases (size depends on DATABASETYPE and DISKQUOTA,
  think 0-50 MB)


What can you do to lower gnunetd's memory footprint?
* reduce GAP routing table size (min is 1024)
* do not compile with debug symbols (-g)
* compile with optimizations (-O2)
* disable internationalization (export LANG=)
* do not load unused application modules (chat, tbench, etc.)


What can you expect to achieve:
* Low footprints are in the area of 2-3 MB (i386)
* Normal footprints are in the range of 10-20 MB
* above 100 MB is a certain sign for memory leaks, try running
  with valgrind and send reports to the developers about
  any leaks that were found; you can have gnunetd automatically
  terminate without a signal using the VALGRIND = SECONDS option
  in section [GNUNETD].  This can be useful to generate a
  valgrind trace for gnunetd that should run over exactly that
  many seconds (typically the longer you can afford to wait,
  the more likely you are to actually observe a relevant event
  in the trace -- and make sure to actually USE gnunetd while
  tracing for leaks).
  One more note about using valgrind.  Since GNUnet uses
  plugins that are unloaded on shutdown, the memory traces
  are likely to be garbeled if nothing is done.  Setting
  the VALGRIND option will disable unloading plugins (see
  src/util/dso.c) and thus improve the traces significantly.


As for memory leaks, so far only one leak in 0.7.0 was discovered, and it had 
a rate of less than 4k per day for most users.  

Christian


On Saturday 17 September 2005 11:36, David Kuehling wrote:
> Hi,
>
> I got a DSL flatrate some month ago, and am now trying to set up a
> permanent GNUnet node using gnunet-0.7.0.  I have stumbled into some
> minor problems (like gnunet-setup -d only saving default seettings), but
> currently the worst problem is that my server almost comes to a halt
> after GNUnet running for 43 hours.  I'm quite new to GNUnet, so maybe
> somebody more experienced can help?
>
> My system is:
>
>    PentiumPro 200MHz, 160 Mb RAM, running Debian Sarge, debianized
>    kernel 2.6.8-2-686
>
> /var/lib/GNUnet is a symlink to a directory on a 40Gb Disk with one
> single partition ("reiserfs format "3.6" with standard journal; using
> ordered data mode"), which is accessed via dm-crypt (aes-128), maximum
> throughput I can achieve (hdparm -t) is 2.7Mb/s due to slow encryption.
> Disk is the only disk on the secondary IDE controller.  No other process
> relies on that disk partition.  I tried enabling unmasq-irq without any
> change.  Kernel uses the deadline I/O scheduler.
>
> hdparm /dev/hdc:
>
> /dev/hdc:
>  multcount    = 16 (on)
>  IO_support   =  3 (32-bit w/sync)
>  unmaskirq    =  1 (on)
>  using_dma    =  1 (on)
>  keepsettings =  0 (off)
>  readonly     =  0 (off)
>  readahead    =  8 (on)
>  geometry     = 65535/16/63, sectors = 80418240, start = 0
>
> hdparm /dev/mapper/cbig
>
> /dev/mapper/cbig:
>  readonly     =  0 (off)
>  readahead    = 256 (on)
>
> (Just tried reducing the "readahead" value, but that doesn't seem to
> help)
>
> GNUnet is configured for SQLite and runs as user "gnunet", group
> "nogroup".  Size of datastore is set to 10Gb.  gnunetd.conf is attached.
> top shows a very high CPU usage:
>
> Cpu(s): 92.7% us,  4.8% sy,  0.0% ni,  0.0% id,  0.7% wa,  0.4% hi,  1.4% s
>
> Although no process is shown that actually uses the CPU that much.
> gnunetd is constantly at 0.0% (and I never saw more usage)!  How's that
> possible?  Permanent forking and killing of processes? :)


> First lines of top (sorted by %CPU):
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  1919 nobody    15   0  3852 1920 3264 S  1.9  1.2   4:26.48 openvpn
> 13369 root      15   0  1512  440 1352 S  1.2  0.3   3:24.54 pppoe
> 22819 root      17   0  2068 1056 1852 R  1.2  0.7   0:01.25 top
> 13857 spock     16   0  5000 1496 4208 S  0.3  0.9   0:01.92 screen
> 22993 root      22   0  3656 2232 2244 R  0.3  1.4   0:00.09 ld
>
> First lines of top (sorted by %MEM):
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>   419 gnunet    16   0  136m  70m 5164 S  0.0 45.4   0:07.94 gnunetd
> 19967 root      17   0 16556  12m  11m S  0.0  7.9   0:11.36 aptitude
> 17199 proxy     16   0  9352 4924 3264 S  0.0  3.1   2:05.79 squid
> 17141 bind      18   0 29980 3044 4248 S  0.0  1.9   0:00.02 named
>  6832 root      16   0  8964 2036 5844 S  0.0  1.3   0:02.05 cupsd
>
> Although gnunet uses horribly much memory, quite enough memory is still
> free:
>
>              total       used       free     shared    buffers     cached
> Mem:        159464     157084       2380          0      16988      29908
> -/+ buffers/cache:     110188      49276
> Swap:        72568      23040      49528
>
> cat /proc/loadavg
>
> 4.24 4.68 4.61 5/101 23500
>
> Opening a man-page takes 30 seconds; running a configure script takes
> more than half an hour.  Luckily, all the NFS-mounted partitions from
> the server are still quite accessible.
>
> GNUnet shows some strange numbers in gnunet-stats:
>
> [..]
> # bytes received via UDP                                    :       
> 266774743 # bytes sent via UDP                                        :    
>    631425234 # bytes dropped by UDP (outgoing)                           : 
>          283328 # bytes received via TCP                                   
> :        223302987 # bytes sent via TCP                                    
>    :        435363595 # bytes dropped by TCP (outgoing)                    
>       :       1123510868 # bytes received via HTTP                         
>          :         26034777 # bytes sent via HTTP                          
>             :         43153134 # bytes dropped by HTTP (outgoing)          
>                :        211916260 [..]
>
> Why are so many bytes dropped via HTTP and TCP (more than actually
> sent)?

Dropped does not mean dropped forever.  Dropped means that they could not be 
send the first time -- and GNUnet will try again later.  Nothing to worry 
about.

> BTW is it a problem for GNUnet that the pppoe internet connection
> reconnects once in 24 hours with the interface getting a new IP address?

Make sure you set HELLOEXPIRES to a value like 60 to ensure that your old 
advertisements expire after 1h.  That should be all that is needed.

> Sorry for the somewhat lenghly mail.  Any help would be appreciated.
>
> regards,
>
> David
> --
> GnuPG public key: http://user.cs.tu-berlin.de/~dvdkhlng/dk.gpg
> Fingerprint: B17A DC95 D293 657B 4205  D016 7DEF 5323 C174 7D40