Re: [Pan-users] Anyone know why pan occasionaly decides it needs to download all headers in a group?

[Jim Henderson]

On Tue, 19 Jun 2012 07:16:05 +0000, Duncan wrote:

> What would make pan try to redownload all headers, instead of just the
> ones since it last downloaded?
> 
> Somehow, that tracking is getting out of sync.

That makes sense.

> The most common way it gets out of sync is if you transfer servers and
> try to use the same sequence numbering, since each server will naturally
> have its own sequencing for a particular group.  This can also happen if
> a server's numbering gets "reset" for some reason or other, maybe
> because it lost its own records and got rebuilt.

Now that is interesting - in my instance, two of the servers 
(forums.opensuse.org and forums.novell.com) point to two separate servers 
that are part of the same "cluster" (the servers are Typhoon servers with 
a Cyclone backend that keeps them in sync - this is the Highwinds NNTP 
software).

While I subscribe to separate groups on each server (and maintain 
different posting profiles for them), the groups are the same groups and 
from what I remember of the Highwinds configuration, the messages are 
supposed to be identical between the two servers, including the headers.

But my larger concern was gmane's groups, as the anti-abuse policies in 
place on their servers prohibit pulling all the headers down (I know this 
because I did it in order to do an analysis on the openSUSE user mailing 
list and got banned for a short time for it).  But now I think of it, I 
haven't seen this behaviour on the gmane server.

My understanding, though, is that pan should be able to handle multiple 
servers that hold the same messages without any difficulty - part of the 
purpose for combining the groups together under one list rather than a 
separate list for each server (which is how it was in "old pan", wasn't 
it?) was so those doing binary scraping could pull missing parts of multi-
part downloads from different servers seamlessly.

At least that's what I recall.  Memory ain't what it used to be, 
though. :)

> You mention that you're syncing between different machines.  If you sync
> newsrcs, but NOT servers.xml, AND you use the default newsrc numbering
> (that is, you didn't rename the newsrcs in servers.xml and the filename
> to reflect the server name instead of an arbitrary number, as I did
> here), it could be that the servers were added in a different order on
> the one machine.  IDR exactly what the default newsrc names are, but
> it's something like newsrc-1, newsrc-2, etc.  Now if newsrc-1
> corresponds to server A on machine 1, but server A has newsrc2 on
> machine 2, then syncing them WITHOUT syncing the servers.xml file that
> maps one to the other as well, will screw you up, because pan will now
> believe the per- server per-group sequential message numbering belonging
> to one server/ group actually belongs to a different one!
> 
> 
> That's one possibility.

Not in my case - I sync the entire .pan2 directory (and the News 
directory) through dropbox.  so the same servers.xml config is used on 
multiple systems, but I've scripted the startup to check for a semaphore 
file I create to see if it's running on another system.  So it's only 
running one place at a time, and I have seen this "download everything" 
behaviour in some cases when I've started pan up only on one system (ie, 
start it on system A, shut it down, start it again, sometimes it'll pick 
a group and start downloading all headers).

> Another would be that something's corrupting or deleting the newsrcs on
> one or more of your machines, such that pan's losing track of which
> messages it has actually seen, and thus when told to download all new,
> downloads all of them.

Now I have seen in a few groups instances where new messages show up when 
I do a Shift+A to get new headers in subscribed groups, but then the 
messages show up as read when I get into the group (and the counter 
updates to reflect this).  Even when it's the first group I've entered.  
I have to manually tweak the newsrc files when that happens (that usually 
happens on gmane groups, though, and not the Novell/openSUSE servers).

> Yet another possiblity, probably remote, is that one or more of your
> servers is having bad enough problems to lose its own message sequencing
> on a group once in awhile.
>
> A variant of that one would be if the server is actually a server farm,
> probably served round-robin, and one of say five or ten servers is bad,
> out of sync with the rest.  Every time you happen to get the bad
> server...  (This one's actually reasonable and I've known it to happen
> with Highwinds based servers, in particular.  One bad front-end in their
> round-robin balancing can be a real frustration!  What can make it worse
> is that nobody likes the bad server, so it always looks as if it has
> less load than the others, and if they're using load stats rather than
> pure round-robin the odds of getting the bad front-end thus go up
> dramatically!)

Hmmmm, now /that/ is interesting.  The two servers in question are, as I 
mentioned, a Highwinds server pair - but they're not served round-robin 
nor in a load-balanced setup. They use separate virtual IPs for their 
different "identities".

> Anyway, maybe you've fixed it by now, maybe not, but those are the
> triggers I came up with in the few minutes I was thinking about it
> writing this post.  If it's not fixed by now, see if any of those match
> what you are seeing...

Nope, it's still happening here occasionally (maybe once every couple of 
days).  The thing that's particularly annoying about it is when it 
happens, I have to stop it and delete all headers in the group, otherwise 
Dropbox ends up trying to sync the cache file.  Some of the groups have 
tens of thousands of messages in them (or even hundreds of thousands), so 
the sync becomes not insignificant over my li'l DSL connection. :)

Jim
-- 
 Jim Henderson
 Please keep on-topic replies on the list so everyone benefits