Re: [Pan-users] Feature request for the git version Pan 0.14.90 and later: Multiserver support for posting to newsgroup.

[Duncan]

Hongyi Zhao posted on Sat, 09 Jan 2016 20:57:12 +0800 as excerpted:

> Hi developer,
> 
> I'm very glad the know the following news on Pan:
> 
> Pan 0.14.90 is the first beta of a ground-up rewrite of Pan in C++.

That is a _very_ old description.  Old-pan, 0.14.x and previous, was 
written in C, but that version hasn't been in development for ... must be 
getting close to a decade, now.

Newer pan, later introduced as 0.90 and continuing thru the present 0.139 
release and 0.140 development/git version, is written in C++.  Of course 
that was while Charles Kerr was still lead dev, as he was from before I 
started using pan 0.11.something, depending on gnome-1, back in late 
2001, thru the C++ rewrite, to 0.132 or so I think.

But at some point Charles found he no longer used news and thus was no 
longer really interested in being lead dev of pan.  He repeatedly asked 
for other devs to come forward if they were interested, but no one did, 
so pan development was effectively abandoned, with only the users and 
distro maintainers providing patches, for several years.

Then KHaley started collecting some of the accumulated patches and his 
github fork became the informal development hub, for those following this 
list at least -- he wasn't interested in becoming an official gnome dev 
and taking over the formal lead dev position.  And, while he did collect 
various patches and did I think provide a few minor patches of his own, 
he didn't do any major pan coding.

Then Heinrich Mueller and Petr Kovar teamed up and brought pan to its 
present form.

Petr is an official gnome dev, but primarily works with translations, not 
code.  As such he has gnome access and in cooperation with Charles Kerr, 
who is still a gnome dev working in other areas, took over formal gnome 
repo maintainership, as well as maintainership of pan's official website 
(as listed in the about box) at http://pan.rebelbase.com .  But, as I 
said, Petr doesn't actually code much, so while he's the official pan 
maintainer now and takes care of the administrative side, he really 
/doesn't/ code much, just coordinating patching, doing releases, managing 
the pan bugs on gnome bugzilla, and managing the official pan website -- 
basically all the administrative stuff.

Heinrich, OTOH, did some MAJOR coding, including introducing some 
features that had been on the wish list for years while Charles was 
lead.  While current pan's general functionality and policies were the 
work of Charles in the C++ rewrite, based on the older C-based pan that 
worked rather differently particularly with regard to multiple server 
functionality but had much the same feel, Heinrich is responsible for 
such major features as attachment uploading and automated actions based 
on scoring.  Tho both had long been discussed under Charles and at one 
point pan even had a (non-functional) GUI stub for uploading attachments, 
Heinrich was the one who finally actually coded up the functionality and 
(based on my description in the case of actions, which was in turn based 
on the last of the old-pan features without a comparable feature in new 
pan, and on previous discussions of how it might work on the list, from 
years earlier) formed the UI by which those features are controlled in 
current pan.

Unfortunately, Heinrich was, I believe, in school at the time, and as so 
often happens to free/libre and open source projects, after he graduated 
and went to work in "Real Life", he no longer had the time to devote to 
pan that he had before, and while he has helped with a few minor update 
patches in the last couple years, there has been no one doing any major 
new feature coding.

On the plus side, while pan definitely has its quirks, with automated 
actions and binary attachment uploading, it's now a reasonably fully 
featured and very functional news client, both for binaries and for text, 
for downloading and uploading, for multiple servers and single server.  
Heinrich finally filled in the glaring holes where major functionality 
was still missing, and pan, for the most part, works extremely well as a 
tool fit for the purpose for which it was designed.  It took decades, but 
pan really is, now, a reasonable approximation of the "Pimp-Ass 
Newsreader", the original PAN name stood for.

> And among all of the new features, I noted the following one:
> 
>   * Multiserver support.  Pan can now download files in parallel not
>   just with multiple connections to one server, but also to other
>   servers.

Indeed, pan has had effectively transparent multi-server support since 
the introduction of the C++ rewrite with 0.90.

> But, I want to request another feature for this and the later version of
> Pan:  Multiserver support for posting to newsgroup.
> 
> Let me describe the background of this feature request:
> 
> Currently, Pan has implemented the multiserver support for downloading
> files in parallel mode just as things mentioned above. But, when we
> posting to a newsgroup via a specific news server, the news server may
> be down at that time and the posting process thus will fail. If we can
> let Pan also support multiserver mode when posting to a newsgroup --
> say, let Pan try all of these news servers one by one, till one of them
> successuflly completed the posting process or let Pan using all of these
> news servers to post to a specific newsgroup, once one of them succeed
> and kill the posting process of all the other news servers.  If so, it
> will be more efficient and convient when using Pan.

In general, the NNTP protocol and news peering agreements among the 
servers is such that posting to only one server is necessary, as the 
servers propagate from there using their peering arrangements.  Indeed, 
posting the same message (as uniquely identified by message-id) to 
multiple servers would result in multiple copies of the same message that 
don't all propagate from the same origin server, and while the protocol 
should handle it, it's definitely not designed for it.  And using 
different message-ids would be far /far/ worse, as that's effectively 
multi-posting the same message as different messages, a form of spamming 
that will get you killfiled by many users and/or your account post-banned 
on decent servers.

That said, in theory at least, a feature that serially (one at a time) 
tried posting to multiple servers, until one server was up and accepted 
the message, at which point pan would stop attempting to post to other 
servers, would be doable.

That, however, is the theory.  In practice, you have to find a developer 
that is sufficiently interested in the feature to actually code it up, 
using a coding style that's sufficiently close to that of current pan 
that it's acceptable to current devs, Petr Kovar obviously, but also 
Heinrich Mueller and KHaley if they're interested in commenting (or 
indeed, in actually coding up the feature in the limited time they might 
have).

The other implication, of course, is that such a tool could potentially  
be easily put to use by spammers, and having pan implicated as a spamming 
tool is obviously not something current /normal/ users are interested in.

Meanwhile... the need for such multi-server-attempt posting in the first 
place is somewhat questionable.  Certainly, various low-reliability 
servers may come and go, but that's a relatively easy problem to solve on 
the poster side, because:

1) Paid server accounts tend to be highly reliable or they don't get 
users resubscribing.

2) Most paid servers don't count uploads against usage limits, because if 
you uploaded elsewhere they'd simply have to download the message from 
elsewhere anyway so they're not saving anything by doing so, and because 
being a server popular with uploaders tends to lead to more downloader 
subscribers, who /do/ pay.

3) It's possible to buy unexpiring "block" accounts, that only count the 
bandwidth you actually use and aren't time limited.

Given the above, it's very possible to buy an unexpiring block account 
for a relatively small fee, on a reliable provider since paid accounts do 
tend to be much higher reliability, and then use that account only for 
uploading, not downloading, so the block is never used up and you have a 
consistently reliable account to upload to.

In fact, various uploaders are known to do just this even if they're 
paying major money for downloading accounts as well, using a separate 
uploading account that they pay only a small, often one-time fee, for, in 
ordered to keep it separate from their download accounts.  Among other 
things, this helps if someone decides to file a copyright claim against 
something they posted, since worst-case, they lose only the relatively 
small fee they paid for the upload account, and can continue downloading 
on their main accounts without interruption.

I don't personally do a lot of uploading (except to text groups like this 
list, which I use as a newsgroup via gmane.org's list2news service), and 
actually don't do much binary downloading either, but a year or two ago I 
did buy a big 1000 GiB $50 block account, unexpiring, from astraweb.com.  
At my rate of usage that will last many years and may actually last my 
lifetime, while it still gives me the ability to download binaries when I 
want to, or when someone has a problem they post here, mentioning a post 
that's giving them problems with pan, letting me check it using pan from 
here.

Their smallest block account, however, is 25 GB, for $10 (US).  So for 
$10, you get access to a reliable news provider for posting, and as long 
as you don't download, you won't use up those 25 GB and you can continue 
using that account as a reliable posting server for years.

Tho astraweb does count header downloads against the block... at a 
discounted rate of 20% -- 10 MB of downloaded headers counts as 2 MB of 
download usage.  I believe to prevent all usage, you'd need to set pan's 
number of connections to 0 to the astraweb server for downloads, then 
increase it to 1 or more for uploads when no downloads are going on.

Alternatively, it's possible to setup multiple pan instances using the 
PAN_HOME environmental variable to point pan at its configuration, with 
the default of course the normal ~/.pan2 location, if the variable is 
unset when pan checks it at loadup.  I actually do this myself, not to 
control bandwidth usage, but rather, to separate my normal text-usage 
groups from my binaries, from test usage (just looking at new groups or 
checking on a post in some random group, that someone here has said is 
causing them problems, etc).  I have three pan wrapper scripts, with each 
one setting a different PAN_HOME (among other things I have the wrapper 
do) before starting pan.

What you'd do for posting control is set your pan download instance to 
post to an invalid server (localhost, 127.0.0.1, is good, assuming you're 
not actually running a news server yourself), just in case, or set it to 
an unreliable server, if you want to try it first.  Then the posting 
instance you'd set to your reliable posting server, but be sure the 
options to download new headers when starting pan or entering group are 
turned off, so they aren't downloaded automatically, and just never 
download them manually.  By then symlinking the posting instance article-
cache to the cache dir of the downloading server (you may want to 
configure both instances for a larger cache, since the default 10 MB 
cache will start purging downloaded messages pretty quickly, potentially 
disrupting things from the posting instance side), and similarly 
symlinking the groups subdir, the newsgroups.* and newsrc files, you 
could start the posting instance and it would already have the headers 
and messages available that you'd downloaded in the other instance.  You 
could then reply to them and post new messages, without downloading 
headers or messages in the posting instance, thereby avoiding downloads 
that will count against that block.

Or simply do what I did and buy a huge 1000 GB block for $50, and don't 
worry too much about downloads... unless of course you do major 
downloading and bust thru 1000 GB in couple months' time and the per-
month, unlimited GB, accounts make more sense.

There's also blocknews.net, which is much cheaper at the low end, 5 GB 
for $2.75, 10 GB for $4.50, and the 25 GB astraweb comparable but for 
only $8.50 instead of the $10 astraweb charges.  However, while they go 
higher, the high end is much more expensive, only 500 GB for $51.49 
instead of the 1000 GB astraweb offers for $50, 1024 GB for $91.39, and 
3072 GiB for 239.99.  The breakover point is about 200 GB for $25.59 
blocknews vs 180 GB for $25, astraweb.  Blocknews also has a somewhat 
different header policy.  While astraweb charges headers at 20% of the 
normal rate, blocknews counts them normally but discounts the entire 
total by 10% to account for header downloads.  (At one point, astraweb at 
least didn't count headers at all, but I guess they had some people using 
them only for header downloads, thus not using up their block but without 
the increased news provider reputation that hosting popular uploaders 
brings, so that didn't last.  Now on both astraweb and blocknews, header 
downloads do count at least something toward your block usage.)

Both providers have pretty much ceased expiring posts (tho of course 
they'll remove copyright violations, etc, upon notification) and are 
continuing to increase retention as time goes on, with about seven years 
(blocknews says 2600+ days, astraweb says 2700+ days) of retention, now.  
Unlike astraweb, blocknews _only_ does blocks, however.  They refer you 
to a different domain (usenetnow) for per-month, download-unlimited 
accounts, tho I'd guess both usenetnow and blocknews use the same backend 
servers.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman