[Pan-users] Re: Intermittent 'Too Many Connections' Error

[Duncan]

mjy posted on Tue, 15 Dec 2009 18:59:08 -0800 as excerpted:

> I'm using Pan 0.132 on Ubuntu 9.04 Jaunty. My news is provided by my
> I.S.P. Dslextreme. I'm allowed 2 connections, and my preferences are set
> to 2 connections. This often works fine.
> 
> But when I am downloading multi-part binaries, I sometimes get an
> intermittent "too many connections" error. From watching the information
> in the task-bar, it appears that what is happening is that Pan is
> opening a third connection to start a new part just as it's finishing
> the previous part. If the overlap is too long, it seems to trigger this
> warning, and if it happens too often the server will kill one of my 2
> connections, leaving one.
> 
> Is it possible that my interpretation of the task bar info is correct?
> 
> Does anyone have any idea of how I can fix this irritating intermittent
> problem?
> 
> Any suggestions will be much appreciated.

I don't believe it's opening up more than two (assuming what you say is 
correct, that you have it set for two), but some news server software is 
known to be slow at closing connections when there's trouble.

What's likely happening is that a connection is dieing due to dropped 
packets, and pan is giving up on the dead connection, closing it (with 
the close process likely having dropped packets as well, thus not closing 
properly), and opening a new one, before the other end has given up on 
and properly closed the dead one.

You can actually test various aspects of this yourself, if you like.  The 
simplest way to test dropped packets would be to install a program called 
tcptraceroute (if you don't already have it installed, it's specialized 
enough tho that most won't have it installed by default), and trace on 
port 119 (the normal NNTP port) to your news server.  You can tell it how 
many packets to send, 3 by default, but I'd try 10 or so, and see what 
the results are.  It's a command-line program. man tcptraceroute once 
installed should give you the manpage.  You'll be interested in the -q 
number of queries option, and will use the host, destination port, and 
packet length parameters, with packet length set to 1200 or so (up to 
1500 but a full 1500 may not work for technical reasons, 1200 is large 
enough, but still small enough it should get thru), so you're reasonably 
emulating the full data packets that would be transmitted during a normal 
news connection.

If you're seeing dropped packets on that, you have issues.

You can also try mtr, alternately matt's traceroute or my traceroute, 
depending on where you look for the answer.  This uses standard UDP trace 
packets so won't be as accurate as TCP and may not work all the way at 
all, but where it works, it's very effective at getting a picture of the 
continuing connection over time, as it stays running for rather longer 
and presents the data in several different (toggleable) formats.  It can 
be built with or without the gtk++ graphics mode support, but can run in 
CLI (command line interface) mode either way.  Your distribution package 
will most likely have graphics mode turned on, however.  You can control 
which way it runs with a command line option.

Testing how well your news server handles opening and closing connections 
gets somewhat more complicated.  Basically, you use telnet and open 
connections by hand, login if necessary, issue a couple commands, maybe 
read a message or two, then close them.  As we're testing how well the 
server handles opening and closing connections, you'd run three separate 
telnet instances.  The first two should get in, the third shouldn't.  
You'd then close one of the first two and immediately try again with 
either that one or the third, and it should now get in.  If the 
connection takes a long time to close, or if your new one can't get in 
immediately, you know there are issues with closing connections and 
opening new ones too fast.  I'm not going to detail the actual telnet 
commands to use here, as they're available in the NNTP RFCs, but there's 
a guy that does this on my ISP (cox cable internet), and posts the 
results from time to time.  If you're interested, I could ask him for 
detailed instructions and post them, but it is very much a manual process 
done using telnet, generally at the command line or whatever, and not 
everybody is interested in doing that sort of low-level testing.

You can also make use of the netstat command, to track how many 
connections your machine (the Linux kernel) thinks it has open.  Again 
I'm not going to go into detail as it's in the manpage, but you'll be 
interested in the number of connections, and their status, to the news 
server.  If you see some in time-wait for more than a few seconds, either 
the other end is being slow at dropping old connections, or the 
connection isn't reliable and packets are being dropped on the way, so 
the two ends aren't communicating properly and old/stale connections are 
staying open for way too long, unused.

Normally, precisely because of troubles such as unreliable connections, 
news servers will be configured to drop idle connections after a period 
of inactivity.  This is usually something like 10-15 minutes, but can be 
as little as 1-5 minutes.  Pan sends keep-alive packets when it's not 
doing anything, to avoid the connection dropping too fast, but most 
servers will still drop a connection if it's only keep-alives, after 
15-30 minutes, but occasionally as little as 5 minutes or so, and 
sometimes not for an hour or so, depending on server configuration.

The biggest problems are usually to be seen on big news server farms with 
separate authentication machines.  It's normally the authentication 
machines that track the number of connections, and won't allow you to 
connect more than the set number.  The problem is that after 
authentication, they're generally out of the loop, as you talk to the 
front-ends that actually serve the data.  Now, if the connection goes 
bad, the front-end you were actually connected to may give up on the 
connection and time you out, but fail to notify the authentication 
machine, so it thinks you're still running a valid connection and won't 
let you start another one.

Actually, on the system my ISP Cox is running (as outsourced from 
HighWinds Media), it's not uncommon for users to get "stuck" connections, 
that the authenticator sees as always on, for months at a time.  
Obviously, these connections are then not available for actual use.  
Fortunately, we're allowed up to four connections each, not the two 
you're stuck with, but they're capped to half a megabit per connection, 
so if one connection is stuck, the available thruput just dropped by 25%, 
from 2 Mbps to 1.5 Mbps -- this on Internet connections now advertised as 
10 Mbps down, so even full speed is barely a fifth of total Internet 
speed, and stuck connections are a big problem for the binary users.  
Fortunately, most users are on DHCP, IP address assigned based on MAC 
address, so for those technical enough to know how to change their MAC 
address, it's possible to get a new IP assigned, hopefully one that 
doesn't have any stuck connections accounted at the news server.  The 
alternative for those with biz connections or who don't know how to 
change their MAC address and thus get a different IP address, is to 
contact address@hidden and have him request that the highwinds-media 
folks reset the connections for that IP address.  Since that can't be 
done over the weekend and typically takes a few hours to a day to process 
anyway, it's obviously better to handle it ones-self with a MAC address 
change, for those who can.

As I said, certain news server software is rather infamous for this sort 
of thing.  Highwinds software, the big commercial company that does the 
"big iron" (usually big Sun servers) news server software, is infamous 
for this.  Their products are named after various types of (high winds) 
storms, tornado, typhoon, hurricane, etc.  You may already know or be 
able to ask what your ISP/NSP uses, or the welcome banner that's normally 
sent, but often not printed by most clients (including pan), so you only 
see it if you telnet in manually, usually gives the software brand and 
version info.

There is other news server software, including stuff like leafnode, that 
runs well on standard x86 (32 or 64-bit) machines, which if an ISP ran, 
they'd probably deploy 10 or more x86 machines to the single Sun server 
running Highwinds, but HighWinds is the only big iron commercial name in 
the business that I know of, so it's a very very common solution to see.  
Which is unfortunate, because it also seems very very temperamental.  
When it is configured just perfectly, it is said to run circles around 
all the competition.  Unfortunately, it's extremely difficult to get and 
keep it configured just perfectly, and as conditions change, what might 
have been a perfect config a couple months ago often isn't so perfect, 
today.  So it's not uncommon to have them working great a few weeks out 
of the year, so-so maybe a month or two at a time, twice a year, and 
having serious customer visible issues of some sort or another, the rest 
of the year -- 6-8 months out of 12, total. =:^(  And of course, the fact 
that the servers are big iron and anything but free, and the highwinds 
software isn't free either, tends to encourage NSPs to skimp a bit on the 
specs and only keep enough hardware running to service the demand 
properly according to the specs -- but those specs are for when it's 
perfectly configured, which it isn't, most of the time, so the machines 
are only running at 50-80% efficiency and aren't handling what the specs 
say they should be handling...

As you can probably tell by now, if it were me, I'd be inclined to throw 
100 or 1000 or 10000 x86 machines running standard FLOSS software, 
leafnode or whatever, at the problem, instead of the relative handful, 
perhaps 5-10, far more individually expensive but supposedly handling far 
more connections as well, big iron Sun machines running highwinds.  But 
that's just me.  For better or for worse, most ISPs and many independent 
NSPs (the ones not running their own custom software) run Highwinds.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman