[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [GNUnet-developers] speed/load problem while inserting files with gn
Re: [GNUnet-developers] speed/load problem while inserting files with gnunet-insert
Wed, 24 Mar 2004 13:51:30 +0900
On Wed, 24 Mar 2004, Christian wrote:
It is clear to me that indexing lets say 100Mb
takes its time anyway so why not do it relaxed
in the background without stressing the system.
Something like reading 100kB and waiting a
second for diskcache and other applications.
Before starting to blame the kernel on this, what datastore
are you using? I know from experience that atleast
gdbm turns tardy when it fills up.
gdbm of course :) I thought about mysql but didnt spend
the time until now. So if i see right, blaming the
kernel would be the "correct" thing cause nobody of us
controlls the filesystem buffer flush cycle.
If it flushes 1Mb in one run with many small blocks
spread out over the FS (lots of head movement)
it gets a system hang.
For a few seconds on my machine.
Perhaps the biggest problem here is that GNUnet datastore
is essentially a random access device for very small blocks.
Even if we'd buffer the blocks in gnunetd for writing,
there still would be no guarantee that something like
gdbm will store them as consequent blocks, and then it'd
just hang or thrash around the moment we flushed the buffer.
Yes i agree with that. Thats why i thought about slowing down
the whole process. So the kernel flushes in the meantime by
timeout, without reaching its buffer high water mark and
I don't think artificial delays sound very good. Rather
this is something the kernel should handle smoothly, imho.
As the one and only pseudoconstructive idea, for now, have you
tried running all the related tasks as 'nice'? Any help?
Yes i gave it a try. (nice -n20) Its just slightly different.
That might be because i now have enough other CPU load
(nearly 1) to break down the indexing process
enough. So it just produces not too much throughput
to really bother the harddrive. It still hangs for a few
seconds now and then.
Maybe i will give mysql a try and see if it changes anything
before going for that slow down again. Even though i saw
that indexing behaviour in other filesharing apps also.
I am just afraid that my machine is not the best configuration
and other weak systems might face the same effects.