Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited)

[Jason Harris]

On Sat, Feb 27, 2010 at 11:26:30AM +0000, Kim Minh Kaplan wrote:
> Jeff Johnson writes:

> No.  Whatever the size of the database the logs start really small.
> Then they grow as operations are committed until the next checkpoint.

Correct.  The logs seem to note a timestamp, LSN, key+old value,
and key+new value.  Thus, changing a 1MB value in a BDB by just
one byte should yield a 2MB log entry, IIRC.

> Once the checkpoint is over the log files can be deleted (but should not
> if you plan to do catastrophic recovery or some form of advanced
> redudancy) and the cycle starts again.  So it is perfectly normal to
> have small amount of log files if you remove unused ones.

Catastrophic in this sense does mean that the shared memory backing
files and/or the DB files themselves have become corrupted.  This
often means "page not found" errors in the DBs themselves.

In such recoveries, the log files are replayed against (merged into)
a snapshot/copy of the DB files, inserting, deleting, and modifying
each and every transaction recorded in each and every log file.

PITR is also possible, in which case all transactions past the
desired date are not applied.

> > My logs (particularly after running
> >     db_checkpoint -1
> >     db_archive -dv
> > are not sufficient to recreate the database in its entirety, just by
> > looking at the size of the files involved.
> 
> This is normal and the expected outcome of db_checkpoint.  After
> db_checkpoint you do not need any log files to recreate the database in
> its entirety, the snapshot is sufficient.

Incorrect.  Either you keep the DB snapshot and ALL log files produced
since then or you can't do a catastrophic recovery as defined by BDB.

If you want to MOVE the log files elsewhere, move the ones db_archive
would delete, and copy the remaining ones (which may change some more,
until db_archive would delete them, at which point they can also be
MOVED and must overwrite the older versions of themselves).

> I use the definition of catastrophic recovery from Berkeley DB's
> manual's chapter "Database and log file archival".  With this I can not
> see any need to plan for catastrophic recovery of the prefix tree as it
> can be constructed from scratch and *must* be kept synchronized with the
> keys database: using a prefix tree that is not exactly the one
> corresponding to the keys database sounds like a recipe for trouble.

Almost.  The PTree, if its BDB files are not physically corrupt,
is continuously (asynchronously) synchronized to ./KDB/key
through ./KDB/time's timestamped contents.  So, if ./PTree/* is
OK and physically older than ./KDB/time, and ./KDB/time has not
had records older than ./PTree/ptree's last update removed, the
PTree DB will update itself correctly via its normal mechanism.

Thus, if one snapshots both the ./KDB and ./PTree DBD databases
at the same time, one shouldn't keep ./PTree's log files since
./PTree/ptree is more correctly and accurately replayed/reconstructed
from ./KDB/time via "sks recon."

> That basically means that you can *not* use the catastrophic recovery
> procedure.

Well, on ./KDB/*, feel free.  On ./PTree/*, "sks pbuild" if you don't
have a suitable snapshot, and never bother with keeping and replaying
./PTree's log files lest it become unsynchronized with ./KDB/key.

> The keys database could use some form of backup procedure.  The command
> "sks dump" is a good one but currently it requires that you stop the
> recon and db process.  One of the SKS server operator mentionned that
> removing Dbenv.RECOVER from keydb.ml works fine and could permit to dump
> the database without interrupting the server.

That was me.

-- 
Jason Harris           |  NIC:  JH329, PGP:  This _is_ PGP-signed, isn't it?
address@hidden _|_ web:  http://keyserver.kjsl.com/~jharris/
          Got photons?   (TM), (C) 2004

pgp0ah8_TjjtO.pgp
Description: PGP signature