[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Fwd: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited)
From: |
Jeff Johnson |
Subject: |
Fwd: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited) |
Date: |
Thu, 25 Feb 2010 09:29:44 -0500 |
I invariably forget to hit Reply-All ... apologies.
Begin forwarded message:
> On Feb 24, 2010, at 6:01 PM, Kim Minh Kaplan wrote:
>
>> Jeff Johnson:
>>
>>> BTW, can someone describe -- even superficially -- what is
>>> being attempted with the PTree store? Any details are welcomed,
>>> I'm not yet able to read OCAML code well enough to recognize
>>> what type of store is being attempted.
>>
>> The use of the prefix tree is described in the two articles Set
>> Reconciliation with Nearly Optimal Communication Complexity and
>> Practical Set Reconciliation. You will find them on the Google code
>> pageĀ¹. The mathematics are quite involved though. It is the
>> information that is needed for two servers to know what keys they need
>> to exchange.
>>
>
> Thank you! I've read through those papers about 3 times now ... another
> month and I'll think I understand.
>
> Note that I'm hoping to use the same mechanism for keeping
> RPM package repositories in sync, not only server <-> server, but
> also server <-> client, using the same efficient gossip protocol.
>
> I have another usage case distributing i18n msgstr's through servers,
> not through traditional PO files as well.
>
> These are the reasons I'm looking at sks-keyserver.
>
>
>>> The PTree deadlock is easily reproduced, and (with db_stat) a
>>> detailed deadlock diagnosis could be attempted.
>>
>> How would that be?
>>
>
> Deadlock diagnosis is described in chapter 11 here:
>
>
> http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/index.html
>
> The db_stat I captured (I have the full file if interested) showed
> a page self-deadlock:
>
> ...
> 800036f7 WRITE 3 HELD ptree page 1213
> 2 READ 1 WAIT ptree page 1213
> ...
>
> which (tenatively) I attribute to either
> 1) a write cursor isn't being closed correctly in pTreeDB.ml
> or (my hunch)
> 2) there's a duplicate key that is stored on the same page
>
> Caveat: these are just very preliminary guesses. I'll dig out a fix
> over the next month while attempting to add DB_INIT_LOCK to pTreeDB.ml.
>
>>> Is there interest in adding DB_INIT_LOCK to the PTree store?
>>
>> While I do not see any practical interest in doing this, the symptom
>> smells like a bug hanging in there that could very well be related to
>> the problem Arnold experienced recently. There would definitely be
>> interest in correcting it.
>>
>
> You are correct that sks-keyserver (as is) will benefit little from
> adding DB_INIT_LOCK.
>
> However, there's a class of failures from running other sks operations
> underneath an active server that likely will benefit from DB_INIT_LOCK imho.
>
> (different topic)
> While you are here ...
>
> As part of recovering from the deadlock, some of the indices were
> damaged (checked using db_verify). The fixup for that is
> db_dump ... | db_load ...
> However, db_dump has no guarantee of preserving all data, its just "best
> effort".
> In my case, the data loss showed up with a DB_PAGE_NOTFOUND, which (because it
> was easiest) led to a full reload of the database from a dump.
>
> None of that was hard or bothersome.
>
> However, DB->associate() to attach a secondary table index to a
> primary key store could have regenerated the secondary index
> if a DB_CREATE flag was passed. That is likely easier maintenance
> than loading from a dump.
>
> Caveat: the secondary -> primary association must be unique for DB->associate
> to be used.
>
> Is there interest in making the maintenance of the secondary indices more
> automatic?
> I noted maybe 3-5 problems in mail archives whose root cause might well have
> been
> data loss in the secondary indices. Removing the secondary index, and adding
> DB_CREATE
> to regenerate when opening, isn't hard to add.
>
> Caveat: With DB->associate, all secondary table adds/deletes are done with
> a callback, so a large amount of the update code would need to be rearranged
> in order to use DB->associate().
>
> 73 de Jeff
- [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jeff Johnson, 2010/02/24
- Fwd: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited),
Jeff Johnson <=
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jeff Johnson, 2010/02/25
- Re: Fwd: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Kim Minh Kaplan, 2010/02/26
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jeff Johnson, 2010/02/26
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Kim Minh Kaplan, 2010/02/26
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jeff Johnson, 2010/02/26
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Kim Minh Kaplan, 2010/02/27
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jeff Johnson, 2010/02/27
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jason Harris, 2010/02/27
- Re: [Sks-devel] Adding DB_INIT_LOCK to sks-keyserver (revisited), Jason Harris, 2010/02/26