[Pan-users] Re: slow as molasses

[Duncan]

Paul Cartwright posted <address@hidden>,
excerpted below,  on Fri, 08 Apr 2005 15:53:36 -0400:

> sorry if this has been beat to death, but I am new here:) I use PAN for
> some binary groups. When I select one of those newsgroups to either
> update, or show the hearders, it just goes out to lunch for 10-5 minutes,
> sometimes it doesn't come back. is there a NICe setting, or something I
> can do to limit the cache, or SOME suggestion to help Pan work faster?? I
> have a Dell desktop with 1.6Ghz processor and 256 megs of ram on a
> broadband connection.
> running SUSE 9.2 and PAN beta 1.14.2.91

OUCH!  PAN (older than CVS builds from the last couple months) simply
doesn't scale very well on large groups, beyond a million or two
overviews (wrongly aka headers), even with half a gig to a gig of memory.
With only a quarter gig of memory, you are probably looking at a swap
storm, possibly with as few as a couple hundred K overviews per group.

There are a few things you can do.  One, limit the overviews in a group at
any one time.  Tell PAN to collect only a few tens of thousands, then
delete them when you are done, and try to keep up so they don't add up too
fast.  Of course, any scoring or filtering you do on the group will
increase the processing time, but get over a couple hundred thousand
overviews and you'll never get to scoring anyway, because the original
processing will bring PAN to its knees.  PAN does have a cache size limit,
but the biggest problem is the overviews, not the actual message bodies,
tho they certainly don't help.  Thus, limit the overviews you d/l and
delete them as soon as you can after processing them, to make most
efficient use of what you have.

Two, buy more memory.  =8^)

Three, if you are decently comfortable building from CVS, try PAN from
CVS.  It's not stable enough yet for a full beta so expect some issues,
and be willing to work with them and report them as necessary, but it
/does/ handle memory **MUCH** better, to the point you should be able to
handle a million or two overviews even on a quarter gig of memory.  We're
looking at summer, probably, before it gets stable enough for a full beta
release, late summer or fall b4 a stable release, with the new code.

Four, consider other options, particularly for your heavy binary groups.
For binaries, I now use klibido.  It's a multi-threaded downloader that
automates downloading from multiple servers at once, which comes in handy
with my ISP, which has three servers, but limits connections to 4 per
server and 384kbps per connection, so I can only do 1.5Mbps with PAN per
server anyway, but can max out my 4Mbps connection when drawing from four
connections to each of the three servers (4.5Mbps total, thus maxing out
my 4Mbps pipe).  It's a far younger application, and doesn't have even the
filtering/scoring abilities PAN does, but after using PAN for awhile, I
was simply amazed at how effectively klibido manages both memory AND 12
connections at a time.  The thing is simply fascinating to watch work,
it's just so efficient!  The big drawback, if you're a Gnome/GTK user and
dislike KDE, is that it's a KDE app and thus requires QT and KDELibs. 
That's fine, here, since KDE is my DE of choice, but it might be a factor
for Gnome-only folks.  http://klibido.sourceforge.net , or some distribs
have it available in their package management system already.  (Gentoo
does, but it's only 0.2.0, and 0.2.2 is out.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html