[Pan-users] Re: Strange issue with missing headers when downloading new headers...and a request

[Duncan]

"Brian Costello" <address@hidden> posted
address@hidden, excerpted
below, on  Tue, 10 Oct 2006 20:59:30 -0700:

> If I remember my NNTP (and I probably don't), each group gave your news
> reader a numeric article number which your client used to determine how
> many new articles were available.  If a news provider's numbers were out
> of sync, the news reader would either re-download headers, or it would
> miss headers. I suppose I could write a program that would open a
> connection to each of the news servers in parallel, request the header
> info on a particular group, and see if they're all the same number - and
> if not, change providers.  I was just curious if anyone had experienced
> this problem with a different provider.

I haven't noticed the problem you mention here, but that's not to say it
doesn't exist.

Your memory of NNTP is pretty good (from my understanding, which
admittedly isn't perfect or complete either one), but for one problem: the
numbers you mention are server-specific, so direct comparison between
servers, unless they are specifically number-synced as a few providers
with multiple servers do, wouldn't be useful at all.

Basically, each server assigns numbers sequentially (per group) to posts
as they come in.  The GROUP command returns low-water and high-water
marks, which are the oldest unexpired and newest posts it knows of, and an
estimate of the number of posts in the group.  The estimate can be MORE
than the number of actual posts but cannot (by RFC spec) be LESS than the
number of posts, and need not be anywhere close to accurate.  As new posts
come in, the high-water-mark will be advanced.  As old posts expire, the
low-water-mark will be advanced.  Do note, however, that it's actually
rather likely that there will be LESS posts than the range indicates, due
to cancels (if honored), post-numbering spam filtering, etc.

I'm not sure if the following is the problem you are seeing or not, nor am
I sure how pan reacts to this, but on some servers, "backfill" is pretty
common.  That is, some of the missing posts in the sequence between
low-water and high-water marks gets filled in.  Forte Agent was one of the
first news clients to recognize and allow for this problem with the
"server creates messages out of order" (or similar wording) checkbox
option they had in their config.  That is in fact one of the big reasons
Agent became popular, as the problem occurred on one of the most popular
news servers at the time (Demon News out of the UK, in fact used by many
US and worldwide users at the time, so it wasn't just a local problem), and
Agent users were about the only folks that didn't have to worry about it
if they had that option checked. (The reason it remained an option was due
to the fact that it /did/ add some tracking overhead, and on the slow
computers of the time...)  I know someone that was personally involved in
working on the problem at an ISP level, living in the UK, at that time.

If a client copes with "backfill", it will detect these posts coming in
below the previous high-water-mark and display them accordingly.  If it
doesn't, as is fairly common since a common solution is simply to track
the previous high-water-mark and consider everything below that already
dealt with, it will only see new posts coming in between the previous
high-water-mark and the new/latest high-water-mark.  As I said, I don't
really know how pan deals with this, but your report fits right in with
the backfill problem, except for Fred's observation that it seems worse
when one is browsing a different group while pan downloads new headers. 
If that's the case and not an observation fluke, it's more likely due to
some sort of race condition or inadequate buffering causing pan to lose
some of the headers.

> Now for the request: I think pan could use some feature that would keep
> multiple copies of Pan from running at the same time.  Admittedly, this is
> a "protect you from yourself" type of mod, but it seems to me that there's
> never a good reason to have two instances of Pan open at the same time. 
> On a different PC, I once started two instances of Pan at different times,
> updated headers for a group on each instance, THEN shut down one instance.
> When I was shutting down my PC, I noticed the other instance, which got
> closed and messed up the header count (at least).  That seemed to screw
> things up (understandably).  Is there something that can be

... you didn't seem to finish the sentence, at least in what I got here,
but the intent of the question is obvious.

With old-pan, I used to run into the same problem, altho I've not run into
it on new-pan yet.  One workaround, provided that you realize you have two
instances running before you have changed much in both, is to shut down
the low-history (normally the second copy) instance first, losing only
what you did in it, before shutting down the changed instance.  As long as
the main instance gets shutdown last, things seem to be fine, altho you do
have to redo anything you did in the other instance.

That said, THERE'S A BIG REASON to allow multiple instances at once now!
New-pan honors the PAN_HOME environmental variable, so you can set up
multiple totally separate configs.  I routinely have my binary instance
(PAN_HOME="~/pan/bin") and my text-group instance (the one I read this
group/list on thru gmane.org, PAN_HOME="~/pan/text") running at the same
time.  Because their configs, caches, etc, are entirely separate, there's
no problem, and in fact it's quite nice to be able to setup some binary
downloads, then go use my text-group instance while the binaries download.

That of course suggests a workable solution to your problem as well.  To
work the multiple instances, I have different scripts set up for each one,
that set the PAN_HOME environmental variable appropriately before starting
pan.  Similarly, it should be fairly easy to create a script that checks
to see if there's another instance running, and refuses to start if so,
perhaps activating it or popping up an error message.  (Activating it
would depend on window manager specific calls, I believe, and you'd need
something like xdialog/kdialog/zenity installed to be able to display an
error dialog, since few folks routinely run pan from the command line.)

Here's a little stub-script that simply aborts if there's a pan process
already running.  Requires bash and the ps executable from the procps
package or compatible.  Name it pan-launcher or some such, set it
executable, and set your regular launch mechanism (hotkey launcher, menu
item, whatever) to use it rather than launch pan directly.  Something more
elegant is as they say, "left as an exercise for the reader." =8^)

#!/bin/bash
ps -C pan || exec pan

If you run pan on MSWormOS, well... all I can say is I prefer the
software I run to be subject to me, not the other way around as
proprietaryware works, so someone else will have to provide a solution
there.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman