[Pan-users] Re: 0.129 - a few minor issues

[SciFi]

On Fri, 11 May 2007 05:16:21 +0000, Duncan wrote:
> 
> "Jim Henderson" <address@hidden>
> posted
> address@hidden,
> excerpted below, on  Thu, 10 May 2007 17:23:18 -0600:
> 
>> Seems to be related to the number of headers in the group - I had
>> forgotten to set the purge for the server, so I had messages going back
>> several years.
>> 
>> Unlike your issue, I'm not seeing the processor utilization spike - it
>> could just be the raw number of headers having to be sorted each time
>> that was throwing it off.
>> 
>> But I'll poke through the open bugs and see if any of the issues I've
>> run into are in there - I'd forgotten that Charles had set this up to
>> track bugs.
>> 
>> Jim
>> 
>> On 5/10/07, SciFi <address@hidden> wrote:
>>> On Thu, 10 May 2007 19:05:38 +0000, Jim Henderson wrote:
>>> >
>>> > One other thing that I had forgotten to mention - the download
>>> > performance is AWESOME (as I knew it was in these newer builds), but
>>> > I've had points where Pan totally seizes (it eventually does come
>>> > back to life, but while it's stuck it's seemingly hung) when
>>> > downloading groups or switching into a group that has many tens of
>>> > thousands of headers in it.
>>>
>>> I opened this bug:
>>> <http://bugzilla.gnome.org/show_bug.cgi?id=430628> There are a few
>>> others depending on exact scenario. Please check bugzilla and chime-in
>>> there on which ones may be affecting you.  It's a sure way to "vote"
>>> for which bugs are, ummm, bugging most ppl.  ;)
>>>
>>> > [...]
> 
> Please don't top post.  It messes up quote context.  There's a reason
> pan warns about it.  Please respect that on the pan list, even if you
> top post elsewhere.
> 
> Old-pan was fully multithreaded.  This increased performance some and
> responsiveness dramatically, but made coding and debugging timing issues
> between the threads incredibly difficult.  In the rewrite, Charles
> decided that the added complexity wasn't worth it in the main, so while
> pan now spins off worker threads in some cases (when first negotiating
> connections, consolidating back to a single thread after they are
> established, and either just added or actively being tested, for
> decoding), the main loop, group updating, and the GUI are all single
> threaded.
> 
> What's happening is that pan is only multi-threaded now where a single
> thread creates a noticeable bottleneck.  The first place that was
> noticed was at initial connection, where it would sometimes take 10-20
> minutes to establish all the available connections and get up to full
> speed.  If one was only downloading a moderate amount, and at full speed
> could have accomplished the entire download in 10-20 minutes, this was
> obviously very frustrating as it could increase the time to completion
> by 50-100%. I was one of the ones that complained about that.  So
> Charles made connection establishment multithreaded, and that bottleneck
> went away.
> 
> Then someone that was saving attachments directly noticed that pan would
> idle the connections sometimes while it decoded and saved big
> attachments (consisting of 50 yencoded parts or so).  Since it's
> generally the download that takes the time, seeing it sit there idle,
> increasing the total time to completion just to do the relatively fast
> local decode and save was very frustrating.  (I didn't see this one, as
> I tend to work in multiple steps, downloading to cache while I do other
> things, then coming back when everything's in local cache to actually
> sort thru things and decide what I want to keep and where, and what I
> want to kill.  Thus the decode and save wasn't occurring at the same
> time as the actual downloading, and couldn't slow the downloading down.)
> 
> As for this new issue, bottlenecking on loading a group...
> 
> * If you have download new headers when entering group set, you'll see
> more of a slowdown.  Try turning that off and triggering your header
> downloads only manually.  That should help.
> 
> * pan now message-threads dynamically as it gets headers, saving the
> threaded list to disk.  This takes FAR less memory as various tricks are
> used to combine similar strings and on multisegment posts, display only
> one header for however many individual segments make up the part.  This
> is more work (and accomplished in that single thread) when downloading
> headers, but where old-pan would do it every time it started (thus the
> minutes long load time if you had many headers to thread), new-pan does
> it once.  Of course, loading the pre-threaded headers still takes some
> CPU and perhaps more importantly disk i/o time.  Since pan now doesn't
> keep everything in memory at once as it used to do, only a single group,
> it reloads this pre-threaded list every time you enter the group.
> 
> * The follow-on from the above is that at least if you have enough
> memory to give the system cache to work with in addition to the loaded
> group, you can decrease the stalls further by loading each group before
> you start your download, then starting your download in one before
> reloading the other group.  pan will still load the pre-threaded message
> list when you reload the group, but it'll already be in cache, so you
> won't have the disk activity to worry about.
> 
> * As mentioned above, I'm not been following the decode-and-save multi-
> threading very closely, as I don't have the issue the way I do things,
> but I'm not sure if it has been merged into trunk or is still being
> tested outside of trunk.  If it hasn't been merged to trunk yet, you'll
> see slowdowns during decode and save, and thus be able to avoid them by
> using my technique of downloading to cache first (while working on
> something else, or sleeping/working/shopping/whatever), then coming back
> when everything's in local cache to do your actual sorting and saving of
> attachments to disk.  You'll need to increase your cache size, but I
> cover that in other posts.

I know you're trying to help Jim.  I just want to say I'm fully
aware of all these issues and am attempting to document repeatable
extensive stalls that won't suit Mac users as a finicky lot.  ;) 
One shouldn't need to resort to workarounds or trickery, but OTOH
do know certain actions will take long times and plan accordingly.

My bug-report also tries to show that a multi-CPU machine is not
being utilised very well at all, but as mentioned in the report I
am wondering if it's intrinsic to glib/gtk/whatever that is
responsible here.  And as mentioned, I have gone through every
component to be sure I've got [p]thread support enabled etc. if
available during its configure.

I see I'll need to file at least two more bug-reports as I need to
document specific areas for each one; it's better to separate the
issues this way.  For ex. the next report I should file is as
follows:  As I type this with an external editor right now, no
downloads can progress, everything under pan2's umbrella is frozen
until I save this tempfile so it can continue.  So the "Post
Article" code needs to have its own thread separate from other
stuff going on.  Of course the workaround would be to let pan2 run
normally, use the external editor by itself to prepare the post,
then when ready, quickly get pan2 to bring up a Post window and
c&p between the two.  Note that pan2's original Post window does
allow downloads to continue, but things get stuck when using an
external editor (even tho that editor itself is multi-threaded). 
This is really driving me crazy how the new download threads are
still forcibly stalled; it should never happen this way if they
are "true" threads.

Charles should've known his single-thread approach in the rewrite
will soon become a major issue with most ppl.  I'm sorry to say
this but foresight should've been part of the rewrite (valour).  ;)

I'd like to have ppl discuss this in the relevant bug-report so we
can show how much & which one(s) are bugging others; even tho I
opened it as a Mac issue, I'm quite sure this is going to include
all platforms.

Thanks for letting me explain all this.