Re: [Sks-devel] 3 million keys & and community help requested

[Jeffrey Johnson]

On Nov 1, 2011, at 2:46 PM, Kim Minh Kaplan wrote:

> John Clizbe wrote:
> 
>> Are we using the most efficient schema for the database? Page sizes and other
>> default DB creation settings and tunables?
> 
> I have started running some benchmarks. First results are available at
> <https://www.kim-minh.com/pub/sks/sks-database-tuning>. The short
> answer is that we are probably not using a good page size.

Apologies for not looking first, you are farther along with db_tuner.

The locking in Berkeley DB is per-page, so most of the (nicely done)
benchmark is likely due to removing lock overhead.

>From some attempts to get strict ACID behavior about a year back,
and some experience with updates failing and lock tunables, I'd suggest
that the Ptree access pattern should be looked at carefully. I've
seen a number of failures attempting to tune with minimal locks
usually when accessing (or initially loading) the PTree store.
The access pattern on the PTree store is more complex than the raw KDB
store.

Again: apologies for not looking at your nicely done benchmark before
commenting.

hth

73 de Jeff