[Pan-users] Re: freeze, high CPU getting new headers

[Duncan]

David Shochat <address@hidden> posted
address@hidden, excerpted below, on 
Fri, 01 Aug 2008 16:39:03 -0700:

> I don't know when this started happening, but it was during the 0.132
> era, so it has to be something that changed on my system. This is a
> fully-updated Ubuntu 8.04 system. I have 1Gb of memory. When getting new
> headers in a large binary group, (either as a result of entering, or
> after invoking Groups->Get Headers...), the progress indicator goes
> normally for a while (10 sec. or so) and then it stops. At this point,
> top says that pan is at 98-100% CPU and the GUI is unresponsive. That
> goes on for as long as 20 minutes, after which it (usually) gets unstuck
> by itself. All other functionality is completely normal. It is only the
> "Get new headers" phase that has this behavior. Another interesting fact
> is that I also can run pan on my MacBook (MacOS X 10.5.4) -- 0.132 --
> haven't yet built 0.133 for the Mac. Anyway, running on the Mac I do not
> have this problem at all. Only on my Linux machine. One experiment I
> tried was to run with PAN_HOME set to a directory on a USB hard disk
> with FAT32 format (I was wondering if it had something to do with my
> hard disk). This made no difference; the behavior was exactly the same.
> Going to 0.133 did not change anything. I'm currently out of ideas, so
> I'm asking for suggestions. -- David

That could be a couple things.

First, make sure you're comparing peaches to peaches (I started to use 
apples, as traditional, but when one /is/ an Apple... =8^) .  Are you 
comparing the same group on both machines?  Suppose there's a message (or 
a regular poster, thus a continuing set of messages) with a bad header or 
something that pan's tripping on.  If you aren't trying to fetch the same 
group in the same general period from the same server (so the set of 
messages is the same), the dataset is different and the test not accurate.

If you are indeed testing the same dataset, then in addition to pan, it 
could well be a problem with one of the libraries, say the i18n 
(internationalization) stuff in (IIRC) pango.  The libraries are likely 
different versions and even if the same, the patches applied by the 
distribution may be different.

Note that the "get new headers" functionality does more than /just/ 
that.  Pan has to get them, of course, but then has to sort them, 
figuring out if and where they attach to current threads, manipulate 
strings such that instead of the same author string appearing repeatedly, 
for instance, it's saved just once and a pointer to it appears in all the 
places it'd normally be (this was one of the big memory saver strategies 
in the rewrite), then after all that, apply the scorefile entries and 
actually update the display.  Part of that relies on various library 
functions, so as they change, so does pan's performance using them.

What I suspect is happening is that, if it's not just the size of the 
group and the number of headers that need processed (which it won't be if 
you did the correct peaches to peaches comparison, as I mentioned), pan's 
getting stuck in a loop somewhere.  This could well be due to a malformed 
header, but it could be a normal one with i18n rules misapplied, too.  If 
it's a malformed header, it could be only on that server or a group of 
servers that got the message from the same propagation, or it could have 
been as posted, so that way everywhere.

The way to trace it would be, most likely, to run pan using a debugger, 
probably gdb, following instructions if necessary from someone with /way/ 
more understanding of debugging C++ than I have.  (I was able to follow 
such instructions as posted by Charles to trace a bug a bit over a year 
ago, and the results did help with the bug, but all I was doing was 
following instructions; I didn't really understand that much about where 
and how to place them, even understanding the idea of a breakpoint and 
etc.)

You /may/ also get results using strace, which doesn't get into the 
internals of the program, but simply reports on activity at the interface 
between the program and the kernel, so you see all the file opens/seeks/
reads/writes/closes, for instance.  I do use it on my own occasionally, 
but be prepared to filter out quite some massive extraneous data looking 
for your needle in the haystack.  If you've never run it before, you'll 
be quite surprised at the number of places the system looks for a library 
before finding it, for instance, and the number of libraries and config 
files even a trivial X program will open and/or (in the case of settings) 
look for and not find.  It's really quite astonishing to realize that's 
going on for every program you run, routinely, and you really do 
appreciate the load times you get!

Meanwhile, switching tracks here...  FWIW I've found it better /not/ to 
fetch overviews/headers automatically upon either pan start or entering a 
group.  I have pan set to just fetch them when I tell it to.  That way, 
if there's a problem such as yours, I don't end up waiting for it to 
clear every time I start pan or enter the affected group.  It'll only 
happen when I trigger it.  Or, if it's a problem loading the group from 
disk, that then becomes clear as it happens separately from actually 
fetching overviews.  Yes, it's a personal preference, but it's one that 
can affect your frustration level when things are going wrong.  Plus I 
just prefer to have the groups stay as they are unless I specifically 
update them.  YMMV as they say.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman