Re: [Sks-devel] sks-peer.spodhuis.org catching back up

[Jeffrey Johnson]

On May 29, 2012, at 10:08 PM, Jason Harris <address@hidden> wrote:

> On Tue, May 29, 2012 at 05:42:03PM -0400, Jeffrey Johnson wrote:
>> On May 29, 2012, at 5:29 PM, Phil Pennock wrote:
>>> On 2012-05-29 at 14:20 -0400, Jeffrey Johnson wrote:
> 
>>> Ah.  Everything I tried was using a dbenv, as the context for opening
>>> the db.  I saw nothing in the docs suggesting that opening a db designed
>>> for use with a dbenv was possible without a dbenv.  *sigh*
> 
> Last time I checked, when you run two processes with DB_RECOVER,
> the existing files (__db_.00{1,2,3,4,5}) are immediately removed by
> the 2nd process using the database.  The 1st process keeps its copies
> of these files open, but both processes believe the backing store
> files (key, keyid, meta, subkeyid, time, tqueue, word) are under
> their their exclusive control and, of course, corruption quickly ensues.
> 

Depends on how dbXY_recover is run: if you add -e, then
the __db* files are preserved, and one has a a coherent
locking scheme (because processes identify locks by
reading __db* files).

Meanwhile, running dbXY_recover on a "live" database
isn't exactly recommended for "production" use.

Without consistent serialization through interprocess
locks, you will surely see "corruption". Removing open
__db* files will lead to two different POV of what locks
are held: the actual in odes and dat blocks are not
free'd until the open fdno is closed.

> Removing DB_RECOVER allows multiple processes to join the environment
> and use the database relatively well, until deadlock occurs.  I had
> tried running db_deadlock with various flags but no success.  When
> deadlock occurred, both processes needed to be killed and db_recover
> needed to be run (manually, of course).  This can be avoided if SKS 
> is modified to use BDB better and retry operations in the event of
> deadlock.
> 

Not sure what you mean by "Removing DB_RECOVER …" but
I suspect that an open without DB_RECOVER needs to be attempted,
which, if/when DB_RUNRECOVER is returned, triggers a second attempt
to open a dbenv with DB_RECOVER set.

> Of course, all processes (sks db, sksclient) would need to use the
> same locking mechanism/flags.
> 

Absolutely: the dbenv MUST be shared so that all processes
have a common serialization.

>>> Yes, it was opinion, covering the state of affairs.  Since SKS is
>>> normally going to run as the *only* user of the DB files, and we know
>>> that even using the "sks dump" command to dump current keys requires
>>> stopping sks, I stand by my opinion that the locking is currently not
>>> what it could be.
> 
> NB:  There is currently NO locking in SKS.
> 

Um … perhaps you mean there is no *exclusive* application locking
which is true: there are certainly page mutexes used internally
(and between processes and depending on how built between threads
using futures)

> SKS normally has only 1 process using each DB/environment, so there is
> no chance of deadlock.  "sks recon" just asks "sks db" what has changed,
> according to the latter's "time" DB, and updates PTree with the new
> key=timestamp, value=hash pairs.
> 

There is certainly the chance of recursive access self deadlocking.
There is also the chance of nested accesses self-deadlocking (which
is a programming error).

>> I'd agree that there is something fishy about how SKS uses BDB in
>> the PTRee store. Nothing that can't be lived with ? but I have seen
>> too many deadlocks in the PTree database for me to believe "correct".
> 
> If the "time" DB timestamps don't monotonically increase, "sks recon"
> "detects" a replay attack, although it is truly self-inflicted.
> 

The problem with VM timing was reported here: I'm still not
sure what the mechanism is (from what I think I know about
Berkeley DB, perhaps wrongly). There's also the chance that
tstamps as data change program flow: but that is application
specific, not database related.


>> Note that a "real" fix is a hugely painful amount of QA on a low-incidence
>> error pathway: the existing incidence of approx 1-2 months between failures
>> is more than acceptable imho.
> 
> Not QA, correcting a bad assumption in the original design.
> 

Its painful triggering reproducers on error pathways is all I'm
trying to say: "works" is far easier QA.

> How many keyservers run NTP, BTW?
> 
>> I can/will confirm the behavior if needed: dynamic state reproducers are
>> all that is difficult.
> 
> Check the inode numbers on the existing and recreated/new
> __db.00{1,2,3,4,5} files.  Note that the 1st process takes no steps
> to use the new files.
> 

Yes: if the in ode numbers are the same, then the content is
identical; if not, then there are processes with different
dbenv's not sharing locks and "corruption" will ensue.

Meanwhile, running dbXY_recover -ev on a "live" database
isn't exactly a "production" code path: ymmv.

73 de Jeff