Re: [Xlog-discussion] xlog-0.9 released

[Nate Bargmann]

* Eric S. Johansson <address@hidden> [2003 Nov 10 15:45 -0600]:
> Joop Stakenborg wrote:
> >A SQL backend would be nice!
> 
> IEEEE NO!  I have spent the past 20 years watching virtually every 
> single project I've been involved with using a database fail miserably. 
>  they have fail and because they take longer to develop, less reliable 
> in day-to-day operation, and need greater hands-on maintenance.
> 
> I am currently trying to set up horde+imp+turba+kronolith for a 
> customer.  Besides the fact that the suite is "deficient" in 
> documentation, its dependencies on databases have cost me God knows how 
> much time.  You need to become a full-fledged DBA to install and run a 
> calendar application. GFD, I swear it isn't a program suite as much as 
> the jobs program for administrators.

I wanted to play with an SQL based log,
http://sourceforge.net/projects/phphamlog and quickly discovered what
you describe.  I even bought an O'Reilly book on the subject and
followed along well so long as the SQL db followed a flat file format.
Trying to abstract several levels of normalization (did I get that
right?) left me with dazed and confused.

I tried to find some layman theory on the SQL RDBMS stuff, but what I
found was either very dense or very expense.  I don't think the SQL
route is a good choice for an application like Xlog.  The dependancy on
a properly configured SQL database most likely kills it for the average
ham who already considers Linux to be beyond his ability already.

> >We would only need xlog for displaying the 
> >data and let the database do the work. I have set up a SQL database a 
> >couple of times, it is not for the beginner. So a starting linux user 
> >could use the simple ascii format, the more experienced user could 
> >connect to a database.
> 
> if you really want a database to work with that's not horrible but still 
> can scale, take a look at sleepy cat.  Otherwise there really isn't any 
> advantage to going with DBM type systems unless they allow you to do the 
> same kind of specialize searching that a fuller feature database does.

When I refered to Berkeley DB earlier, I was in fact refering to
Sleepycat who are the present Berkeley DB maintainers.

> In any case, tell me what kind of data relationships you are interested 
> in and I will see if I can think of some alternative structure that 
> won't be too ugly.

Well, I think some natural keys are:

Date/time
Callsign
QTH
Locator
QSL received
QSL sent
Band/frequency
Mode
TX location/callsign

Looking at http://www.sleepycat.com/products/featurelist.shtml it
appears that a few db formats are available.

My initial thought is that the db would be a simple mirror of the flat
file format now in use.  The advantage is that the db could be
structured for easy key/value searches with, I suspect, simpler code on
Xlog.  I envision multiple log dbs still allowed as with the current
flat file format.  I'm just looking for a way to make it easier for
ordered searches to be displayed and printed from Xlog.

> >The major benefit of using SQL is the possibility of using big logs with 
> >10.000+ records. On the other hand, we would still need the slow gtk+ 
> >list widget for displaying the data. So we need to re-think about how to 
> >display the data when using SQL. Maybe we need to design our own widget 
> >for this or wait until the next gtk+ version, which will probably have 
> >better support for displaying large amounts of data in a list.

According to the Sleepycat link above, its db allows databases up to 256
terabytes and keys and values up to 4 gigabyte.  I think this is
probably sufficient for all but the largest contest logs.  :-)

> if you're having a problem with large list widgets, you'll need to 
> rethink your design no matter what.  A technique I've used relatively 
> successfully is the sliding window approach.  Generate a list of keys, 
> position yourself to the right spot on the list and then start fetching 
> records for the visible keys plus a couple screens on both sides of the 
> scrolling box.  the only time you need to update that list is when the 
> date of the file changes relative to the date of the list creation. 
> When you approach within one-half screen of an end on either side, start 
> fetching records for the next screen+.
> 
> is really depends on how fast one can fetch records and how much 
> background retrieval will cost in terms of screen updates.  Sometimes 
> you need to increase the size of the window as well, change trigger 
> points and rate of retrieval of records.
> 
> if you really want to get fancy, you can look at velocity of movement 
> and use that to choose the size of the array fetched.  I recommend 
> starting simple if you should choose to move to this technique and get 
> the basics down etc.

Erp!

That's a mouthfull, Eric.  I think you are demonstrating a knowledge of
this area rather well!

73, de Nate >>

-- 
 Wireless | Amateur Radio Station N0NB          |  Successfully Microsoft
 Internet | address@hidden               | free since January 1998.
 Location | Bremen, Kansas USA EM19ov           |  "Debian, the choice of
  Amateur radio exams; ham radio; Linux info @  |     a GNU generation!"
             http://www.qsl.net/n0nb/           |   http://www.debian.org