[Pan-users] Re: Increasing connections to server decreases overall speed

[Duncan]

walt <address@hidden> posted
address@hidden, excerpted below, on  Fri, 03 Oct 2008 23:40:22
+0000:

> On Fri, 03 Oct 2008 15:30:05 -0700, Rick Barry wrote:
> 
>> I'm running Pan 0.132 on Ubuntu 8.04.1.
>> 
>> I modified the number of connections in the servers.xml file from 4 to
>> 10 which is how many my news provider allows on my account.
>> 
>> I notice though, that as connections start up, the speed drops from
>> about 400 kB/s to just under 330 kB/s as it approaches 10 connections.
>> Is this a function of my ISP pipe (DSL) being too small, or am I just
>> seeing less speed but greater throughput due to the larger number of
>> connections?
> 
> First, how are you measuring the total throughput?  The bottom left box
> of pan's status bar?  I've never looked at the code to find out how that
> calculation is done, and so I can't tell how accurate it is or isn't.

IIRC (I haven't actually done binaries in awhile, now, and text doesn't 
use enough speed to matter), pan's fairly accurate.  I run a ksysguard 
panel with a bunch of system status graphs including upstream and 
downstream bandwidth, and pan's estimate wasn't too far off.

> Second, what is the advantage of using 10 connections?  This has puzzled
> me for a long time and I've heard several different explanations which
> haven't quite convinced me yet.  This question always sparks a lively
> debate, which I've already heard several times but wouldn't mind hearing
> again.

The biggest reason for multiple connections is when the server caps the 
per-connection speed to something below your general Internet pipe 
speed.  If it then allows more connections than you are using and you 
still have pipe bandwidth to spare, you might as well boost those 
connections up in ordered to make use of all the bandwidth you can.

If the individual connections don't have a cap or it's higher than your 
general Internet pipe cap, or if the sum of the allowed connection caps 
is higher than your pipe cap, then using connections beyond some level 
simply uses resources unnecessarily, and as apparently the case here, may 
actually slow down the total effective speed.  However, one can't really 
name a point at which that's the case for all configurations, because the 
location and restriction of the bottleneck varies.  

If the machine is an older single-core with a single hard drive spindle 
and say half a gig of memory, it's reasonably likely that it simply can't 
keep up with more than say four threads effectively, and may have trouble 
with more than two.

OTOH, on my main machine, a dual dual-core Opteron 290 (2.8 GHz), with 8 
gigs RAM, with 4-spindle kernel/md RAID-6, I could probably handle 
something higher, say 8 connections, if the available bandwidth was 
sufficient to warrant it.  Unless the server was per connection capping, 
however, and I needed more connections to fill my bandwidth due to that,  
I really doubt upping that to 10 connections would bring me much more and 
it may still actually slow me down, simply due to the increased overhead.

Besides fiddling with the number of connections, it may also be worth 
considering setting cache size appropriately (preferences.xml) and 
downloading to cache to go thru in a sort-n-save session later after 
they're all downloaded, rather than decoding and saving attachments 
directly.  This is what I do.  That puts less stress on the local system 
during the download as it's not having to decode and save the attachments 
at the same time, and is only downloading the raw messages straight to 
cache.  It's also far less frustrating to me, because I can set pan up to 
do the download, then do something else (go to work, sleep), coming back 
latter to sort things out.  That way I'm not trying to view stuff while 
pan's still downloading it and having to wait to do so, and when I come 
back, I can sort and save the files (immediately, no waiting to download) 
to their final destination while I still know what messages they were 
attached to (so have the subject, the date, the sender, and any 
accompanying series description message, all available) and can use that 
info in choosing where to put them.  The caveat is configuring a large 
enough cache, however, as pan's default 10 MB just doesn't cut it if 
you're doing that with binaries.  I remember years ago when I first 
started using pan, before I figured out the cache size thing, getting 
frustrated because I'd setup a download to cache, watch the first few 
download then let it do its thing, only to come back later and find it 
had downloaded and then deleted most of the stuff before I even got a 
chance to look at it, because the cache was only set big enough for maybe 
a tenth of what I had told pan to download!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman