Re: [Sks-devel] sks (fast)build memory/cache problem (was: sks.pkqs.net will be back again)

[Stephan Beyer]

Hi,

On 30.06.2012 13:31, Stephan Beyer wrote:
> The DB is already building based on my last keydump.

No, it was not. The process got stuck after a few minutes or seconds.
In fact, neither build nor fastbuild works.
I already "hg bisect"ed the sources and it did not work with any
version that compiles on my system.

What happens? sks hangs (deadlocks) at futex(), says strace.
Interestingly, it seems to depend on the cache size:

$ sks build -n 2 -cache 56 dump/sks-dump-000{1,2}.pgp
Loading keys...done
DB time:  0.04 min.  Total time: 0.04 min.

works. However, -cache 55 does not work. This is reproducible.

For 3 dump files (and -n 3), I have to use -cache 87;
for 4 (and -n 4), it's 117. For 4 and -n 2, even 119 MiB
are needed.
If the -cache requirements are climbing this way, I will
not be able to build the DB by memory restrictions...

For fastbuild, it is even worse (119 MiB cache do not suffice
for 2 files).

What if I do it step by step?

$ sks build -n 1 -cache 60 dump/sks-dump-0001.pgp
(works)
$ sks merge -n 1 -cache 60 dump/sks-dump-0002.pgp
(works)
$ strace sks merge -n 1 -cache 60 dump/sks-dump-0003.pgp
...
futex(0x7fc9a4188370, FUTEX_WAIT, 2, NULL

Interestingly, if I take a look into merge.log now, I can see:

2012-07-01 04:13:03 Fatal database error: Bdb.DBError("BDB2034 unable to
allocate memory for mutex; resize mutex region")
2012-07-01 04:13:03 closing database...

So it is really a memory/-cache issue.

Any ideas how to build the DB on a "low" memory system?
Without -cache, it does not work, too, btw.

Best
  Stephan

PS:
 sks worked all the time on a 1 GiB system (much less free), and
 the DB built without any problems in 2007 (where the keydump was a
 little smaller, indeed).