Re: [Sks-devel] sks-peer.spodhuis.org catching back up

[Jeffrey Johnson]

On May 29, 2012, at 1:49 PM, Phil Pennock wrote:

> Kristian drew my attention to sks.spodhuis.org showing a last update of
> May 25th.
> 
> I'd tried to make sksclient be read-only in BDB access, but failed to
> find a way to do so because a directory-based BDB *always* seems to use
> log files.  So I'd given up and gone with the flow.
> 

Sure there's a way to use Berkeley DB without transactional logs.

FWIW, what is actually implemented in the SKS key servers isn't fully ACID:
I can dig out details if anyone really cares. I looked a bit at extending the
implementation to be fully ACID, but stopped because of the difficulty in
identifying the records being retrieved because OCAML isn't my favorite 
language.

There are other modes of operation for Berkeley DB that do not need
transactional logs, nor the fine-grained per-page locks. E.g. the DB_INIT_CDB
model does per-database locks and still guarantees that active
exclusive writes will not interfere with multiple shared readers. One
might need to worry about writer starvation but there are solutions there too.

Its also possible to open the databases RDONLY without a dbenv to access 
information
independently of the actual running daemon, without locks, and without
guarantees that the client won't occasionally (and its _VERY__ occasionally
because of the sheer number of records and accesses) if a client attempts
a read simultaneously with a server attempting a write. (a trickle writer
to move cached data in __db* files into the backing store might be needed
here: its not hard coding even if difficult QA).

> Apparently the locking is not what it could be, as I ended up with a DB
> that needed recovery.  Moral 1: do not run sksclient against a serving
> DB.
> 

There's no engineering diagnosis in your statement that indicates anything
other than opinion with
        … the locking is not what it could be, …

The recovery was likely needed for other reasons. Note that modern Berkeley DB
supplies DB_RUNRECOVERY during open and its not too hard to automate the 
recovery
as part of daemon startup, rather than forcing manual intervention.

Adding lock registration by setting this in DB_CONFIG is also useful
        set_thread_count 64
If/when you attempt that, you will need to run
        dbXY_recover -v
so that the the __db* files are recreated with space for the lock registration.

If the locks are registered, it starts to become possible to do a post-mortem
for locking issues, and its also easy (with kill(2)) to tell whether locks are 
"stale"
(as in a reader process died before releasing a lock) or "live" (i.e. some 
running process
still has an open dbenv).

hth

73 de Jeff