Re: [Pan-users] Re: Pan and the 2.6 kernels

[Rich]

Hi,

Thanks for your detailed response.  I thought it may have been something
on my end that I overlooked.

I was thinking about the way the listboxes are handled in GTK.

I used some Data Widgets for VB a while back. We ran into the same
issue, and the solution was using a virtual list.  Where the only list
items shown were the visible number of lines plus two (previous and next
items) when the list was scrolled.  I was able to store 50,000 line
entries without sucking up valuable Win95 memory resources. :)

It worked on the same premise, where all the list items would be in a
database.  Would keep track of the database cursor for prev (current -
1) and next (current + viewable rows ) testing for EOF or BOF as well to
stop scrolling.

I was the new maintainer to see if IT could be fixed.  The load time
took about 10 minutes, and ran out of memory most of the time. With the
new approach, the app loaded in a few seconds - with access to all items
in a fraction of the time and space.

Good luck on the rewrite.

I found if I whack the articles before downloading the next mega
bunch... pan works okay.  It seems its dying on marking old articles -
changing from Bold to thin (or greyed).

Thanks for the other news app - I will check it out.
Hope to see the new improved pan soon.  :)  

Regards,
Rich






On Sun, 2005-01-16 at 05:19 -0700, Duncan wrote:
> Rich posted <address@hidden>, excerpted below,  on Sat, 15
> Jan 2005 12:11:05 -0400:
> 
> > I've been using the 2.6 kernels for a few months now. Started with 2.6.7,
> > and have recently installed 2.6.10, which is MUCH faster and appears to
> > have had some issues addressed.
> > 
> > However, pan runs somewhat better in 2.6.10, but still grinds the system
> > to a standstill for seconds at a time.  It used to be minutes with prior
> > kernel.
> > 
> >>From what I can see, if I delete all the articles in pan, and then get
> > new articles. I do not see the system halted. When I do not delete the
> > articles and get new, the systems starts processes spike and everything
> > freezes. Sometimes I was able to get at a text prompt and kill pan, which
> > the system them resumed as normal.
> > 
> > It seems to happen more on the larger newsgroups.
> > 
> > Is there anything I can do to stop the system hangs?
> 
> In general, no.  PAN is a resource hog on large groups, due to the way it
> handles things.  If you have less than a gig of RAM, PAN will likely be
> swapping like mad on large groups, as it tries desperately to get enough
> memory to chew thru all those overviews (often incorrectly called
> headers, incorrect, because the overviews are only a limited subset of
> the actual message headers) it's attempting to sort.
> 
> The reason 2.6 works better than 2.4 is because 2.6 manages scheduling and
> multi-tasking better, particularly under heavy load such as when there's a
> process eating up all available real memory and throwing the system deep
> into swap, as PAN will be doing under these circumstances.  While it /is/
> possible to tweak the 2.6 scheduler and swap mechanisms, and that would
> possibly get you slightly better performance, the real problem is that PAN
> is using GTK+ widgets that simply weren't designed to scale to hundreds of
> thousands or even millions of entries, and at present, PAN doesn't really
> make it much easier for them, as it keeps everything in memory (altho it's
> /far/ better than it was, by about a factor of 10, managing a million or
> so overviews as opposed to the 100k it could manage before, without
> choking, on a 1/2-3/4 gig memory system).
> 
> That's why Charles and others are currently working on switching PAN to
> use a real database back-end library (SQLite), as it should scale far
> better than the current widgets, which while they have some data
> functionality were really intended for display, and simply don't scale
> well in data mode.  As well, while they are doing the back-end db rewrite,
> they are making additional memory saving improvements, such as only saving
> the common elements of a subject line once, then using a much smaller (4
> byte, 32-bit, on x86_32) token to refer to it for each message with the
> same general subject line, instead of saving the entire thing once for
> /each/ message or message-part, as PAN does currently.  Another trick they
> are using is only keeping in actual memory the data for a few thousand
> overviews at a time, using new and more efficient database to store the
> rest of the overviews for the group on disk.  Thus, when it's all done,
> PAN's scaling should be virtually unlimited, while instead of using
> hundreds of megabytes of memory and /still/ running out of resources on
> the largest groups, it'll only use tens of megabytes of memory, and
> continue to be responsive (both it and the system) even with groups with
> millions or tens of millions of overviews.
> 
> However, that's a fairly major rewrite of the back-end code, and isn't an
> instant process.  Depending on how much time Charles has to integrate the
> work others have already been experimenting with, I wouldn't expect a beta
> of this to be out until late Q2 or Q3 at the earliest, and that's likely
> to have a significant number of bugs still.  We may be looking at Q42005
> or into 2006 before it's stable, even if there's reasonable time for
> development.  However, it /is/ being worked on, now, which is good as that
> particular project has been put off for some time, because it /is/ such a
> major project.
> 
> In the mean time, if you haven't yet, look up Klibido, a
> new-this-past-summer KDE binary news harvester application.  Being so new,
> its interface is rather rough still, and it lacks features such as
> automated filtering/scoring entirely, but it's extremely efficient at what
> it /does/ do, manage multiple connections to multiple servers (if you have
> them), automatically downloading parts from whatever server they are found
> on, sucking down news binaries with a speed and overall system resource
> efficiency that frankly left me (and some others I've talked to about it)
> in awe.  It is KDE based so you'll need to have at least the KDE basics
> (qt, kde-libs, kde-base) installed, which might be a negative for some,
> and as I said, it's new and obviously unfinished and a bit rough, but the
> thing works with an efficiency I simply hadn't realized was possible! 
> Being so new, your distribution likely won't have it except in
> testing/unstable, but there are various types of packages and a tarball
> for those who wish to or need to compile it manually, on the site. 
> (Gentoo, my distrib of choice, already has it in portage, altho it is
> still keyworded ~arch, which means unstable.)
> 
> http://klibido.sourceforge.net/
> 
-- 
Rich <address@hidden>