Re: [Pan-users] Making Multiple Servers Happen In Your Lifetime

[Eric Ortega]

On 7 Oct 2002, Mark H. Kraml wrote:

> If we keep talking about this feature, we may get some good ideas down
> and actual traction on the feature may follow.

I agree that it's worth a shot, it's just that this particular "feature"
introduces many additional implementation complications which may not be
obvious even with some _thinking_.


> I just signed up for a premium news account, clearly this feature would
> be of great benefit. After giving it some thought it is, as stated
> earlier, not so simple to do.

Yes.


> A least not without a good hit on performance.

Actually, allowing for a massive hit on performance makes it a lot easier
"to do".  :)


> While my regular ISP has a limited amount of storage for
> news (about 1-2 days), the premium ISP has over 20 days of storage. This
> makes some newsgroups that have over 1 million headers.

Yes.


> It take a long time to download "ALL" headers from there. They also give
> me a 6GB limit for 30 days, this means I really only want to get from
> there if I really have to.

Yes.

To provide some "traction" I would like to bring up some of the discussion
that we had WRT this topic many moons ago:



  * People connect to the internet differently:

    - Some people have high-bandwidth:

      : some people have high-bandwidth which charges per-byte with "no
        limits" on rate

        ; some of the per-byte charges are only instantiated for "out of
          network"
        ; some of the per-byte charges are through ISP's newsservers (as
          applicable)

      : some people have high-bandwidth flat-rate with limits on rate

        ; some have access to a high-bandwidth newsserver within their ISP's
          network which may or may not suck
        ; some simply want to use their high-bandwidth in the best manner
          possible without subscribing to anything premium ... this means
          combining across multiple (as in stupid many) free newsservers
        ; some want to download as much as possible from a "free" (as in
          ISP provided) server before a premium

          ' This (possibly) implies on-the-fly decisions about server
            response times.
          ' This (definitely) requires a server download priority system.
          ' This also (definitely) requires a fuzzy intelligence about
            download rates +- concurrent connections +- current response.

      : some people are a mix (and they're just going to have to figure
        things out for themselves :)


    - Some (many) people have low-bandwidth

      : some of these users may want to combine across free servers as in
        the high-bandwidth case for whatever reason

      : some of these users may want to use their ISP's servers
        predominately with a combination of free servers to fill in lost
        posts

      : these users are usually interested in collecting posts overnight

        ; this means automated choosing between "best servers"
        ; this means decisions based on current response times


  * People use newsservers for different purposes:

    - Some people are interested in text:

       : these people want mixtures of servers in order to archive for as
         long as possible (disregarding dejanews.com)
       : these people are not as concerned about fastest rates of transfer
         as opposed to ensuring parent posts

    - Some people are interested in binaries:

       : some want the best combination of free servers to ensure free
         downloading of multipart posts
       : some want fastest first while disregarding cost
       : some want cost first, speed second


  * People/users are sometimes dumb/naive

    - some people ask for something whether or not they have really thought
      about whether they were going to use it that often
    - some people think they will use something and then realize that they
      don't
    - some people realize it's easy to ask for something, much easier than
      actually doing it themselves, and fall into the category above
    - without programming experience it is sometimes difficult to
      understand what a request entails ... this causes many problems WRT
      programmers and management (see http://www.slashdot.org for
      examples).



Please allow me to demonstrate my current usage of Usenet in order to have
Pan optimized for ME.  :)

  I am most interested in a clear progression of threads within groups.
    This appears to be well done at the moment.

  I, secondarily, download veritable assloads of porn, music, etc.  In fact,
    I am doing so RIGHT NOW!

  I have two newsservers which are immediately available, one from the ISP
    and one for which I pay extra.  The newsserver from the ISP limits
    connection speeds to 18kB/s, but unlimited connections.  The premium
    allows for fastest/unlimited but charges 6 Gb/30 days.


As a consequence of my "physical" limitations (as concerns
bandwidth/concurrent connections), I would prefer to have the following
happen:

  A single Pan session (currently I run 2: one for binaries, one for text).
  When I click on "update headers" I get 6 concurrent connections, four to
    my ISP and two to Easynews (the premium newsserver).
  All headers are mixed appropriately to give me the longest possible
    threads and most complete binaries.  (Reminds me of the song which
    goes, "Dreeeaeeamin' DreamDream")
  When I am reading responses to a group I am unaware of which particular
    server the responses/posts are being pulled from, but I can specify
    to which my posts will go.
  When I download binaries I am able to specify at least a simple cost
    function.  It could even be as simple as "Always download from here
    with 4 concurrent connections unless you absolutely have to download
    from elsewhere."


This, of course, brings to mind the suggestions of "ratemeters" for certain
newsservers and the like.


> On the topic of performance, I have a 1GHz P3 with 512MB RAM, not the
> fastest round, but not out of the norm of machines we would want to
> support. Even today, with 1 million headers, the performance for loading
> and managing groups is quite, well, unimpressive.

I agree, but I use a dual PII 300MHz machine for the majority of my Pan
usage.  This machine does CD-burning, Pan-through-VNC downloading,
mail/web/ftp, etc. and is extremely passable due to the efforts of the Pan
development team.