Re: [Sks-devel] Issue Importing Database Dump (Dents in my forehead)

[John Clizbe]

Brian D Heaton wrote:
> Howdy,  I'm Brian and I'm trying to get a keyserver running on a
> Rackspace hosted cloud server.  Reading the archives about brokenness in
> the Ubuntu packages, I've compiled both BDB and SKS from source. 
> Attempting the full-build, the system deadlocks after reading in the 3rd
> of 4th dump file.  I'm able to build the DB with a single dump file
> (testing).  If I then manually do the cleandb and pbuild, those steps
> succeed.    I then attempt to do a merge one at a time with the
> remaining dump files, by the 3rd or 4th file, I get a mutex error.  I've
> provided a lot of information below based on my reading of the archives
> and the questions usually asked when someone posts an issue.
> 
> I'm sure I'm missing a simple option somewhere.  I'm open to hints,
> whacks over the head, or even simple solutions. :)
> 
> THX-Brian
> ---------------------------------------------------------
> 
> OS: Ubuntu 12.04 (LTS)
> ARCH:  X86-AMD64
> RAM: 1GB
> BDB Version: 5.3.21
> SKS Version: 1.1.3 and 1.1.4 (cloned repository)

¡Hola, Brian!

I'd suggest going with 1.1.3 until you get a working install. If you must
clone a dev repo, stick with Yaron's (trunk), it's saner than Kristian's or
mine. We both (usually) test things before committing and pushing, but I
wouldn't recommend the dev repos until you're up and running. Another VERY
import thing you should remember, the SKS source currently is not LSB FHS
aware. Everything is expected in one directory, typically /var/sks.
> 
> BDB Detail:
> -- Configured with with pthread and posixmutex.  I've also built it
> without posixmutex with no change.
> -- Confirmed that sks is linked against libdb-5.3.so via ldd.
> -- Confirmed that only libraries and headers for v5.3 are present.

I maintain SlackBuilds for everything past 4.4, For db53 I used
../dist/configure \
  --enable-shared --enable-cxx --enable-compat185 --enable-sql \
  --enable-sql_compat --enable-amalgamation --enable-sql_codegen

Solaris' OpenCSW freeware project uses
  --enable-compat185 --enable-o_direct --enable-cxx --enable-java

MacPorts defaults to --enable-cxx with java, sql & tcl as options

The Mac Fink project uses:
  --disable-static --enable-compat185 --enable-dump185 --enable-cxx \
  --enable-stl --enable-sql --enable-sql_codegen

Hint: Check what Ubuntu use[d/s] and stick as close to that as you can.

> Output of initial build:
> ----
> address@hidden:/var/lib/sks# sks build /var/lib/sks/dump/sks-dump-0001.pgp
> -debug -debuglevel 10 -stdoutlog -cache 100

For 1GB of RAM, you should be safe with -n 7 with build. When I built on 1GB,
I ran into swapping with n >= 8.

> 2012-06-16 04:42:35 Opening KeyDB database
> Loading keys...done
> DB time:  0.03 min.  Total time: 0.04 min.
> ---
> 
> Output of PTree Build:
> ---
> address@hidden:/var/lib/sks# sks pbuild -cache 20 -ptree_cache 70
> -diskptree -debug -debuglevel 10 -stdoutlog
> 2012-06-16 04:45:21 Opening PTree database
> 2012-06-16 04:45:22 Opening dbs...
> 2012-06-16 04:45:22 Opening KeyDB database
> 2012-06-16 04:45:22 5000 hashes processed
> 2012-06-16 04:45:22 10000 hashes processed
> 2012-06-16 04:45:23 15000 hashes processed
> 2012-06-16 04:45:23 Cleaning Tree.
> ---
> 
> Failing output of merge:
> --
> sks merge /var/lib/sks/dump/sks-dump-0002.pgp -debug -debuglevel 10
> -stdoutlog -cache 100
> ...
> ....
> 2012-06-16 04:48:35 Adding hash 03C0D889F24E3D48DB6B0572504947F3
> 2012-06-16 04:48:36 Fatal database error: Bdb.DBError("BDB2034 unable to
> allocate memory for mutex; resize mutex region")
> 2012-06-16 04:48:36 closing database...

Never been a fan of merge myself. I think there are other better ways around
this than merging in a single file at a time (read on). But this error leads
me to suggest db_recover, first without -e, then a second time with -e to
recreate the DB environment space AFTER putting in place a sane DB_CONFIG.

  db53_recover  -h /var/sks/KDB
  db53_recover -eh /var/sks/KDB

That's how regions get resized. (or just run the first and then sks will
recreate the regions)

> DB_CONFIG file present in both KDB and PTree directories:
> Note:  Yes, I got frustrated and cranked up the mutex_set_max.  Looking
> at "dbstat -x" it doesn't appear mutexes are being released between
> merge runs.
> ----
> set_mp_mmapsize         268435456
> set_cachesize    0      134217728 1
> set_flags               DB_LOG_AUTOREMOVE
> set_lg_regionmax        1048576
> set_lg_max              10485760
> set_lg_bsize            2097152
> set_lk_detect           DB_LOCK_DEFAULT
> set_tmp_dir             /tmp
> set_lock_timeout        1000
> set_txn_timeout         1000
> mutex_set_max           655360000
> set_mp_mmapsize         16777216

You have two values for set_mp_mmapsize. A rule of thumb is ~ 1/4 avail
memory. mmap can speed performance on reads. Pick one size, OK?

I've never exceeded an inuse mutex count of ~42k, you shouldn't need that high
of a number either. Tunables in DB_Config won't help you on build, the BDB env
isn't created until 'sks clean'. Likewise the environment isn't created in
PTree until it's used after 'sks pbuild'.

I /have/ seen things hang while building the DB, but it has always been in
pbuild for me, never build. The reason it's happened for me is that Berkeley
DB takes out a mutex lock /per page/. Small pagesize ==> more locks required.
Thus a Catch-22, since you can't have a directory containing DB_CONFIG in
place before you create the DB.

Do you have sksconf present in the same directory you are in when running 'sks
build' (and clean, etc...)? Do you set pagesize and ptree_pagesize?
If not, I would recommend setting the following two in sksbuild:

  pagesize:       128
  ptree_pagesize: 16

They'll both do you more good when building than mucking about with DB_CONFIG.

Good luck,

-John

-- 
John P. Clizbe                      Inet: John (a) Gingerbear DAWT net
SKS/Enigmail/PGP-EKP                  or: John ( @ ) Enigmail DAWT net
FSF Assoc #995 / FSFE Fellow #1797  hkp://keyserver.gingerbear.net  or
     mailto:address@hidden

                   Cowboy Haiku -- Reflections on Rodeo
So many Cowboys. / Round Wrangler butts drive me nuts. / Never enough rope.