Re: [Mldonkey-users] New source management

[Goswin Brederlow]

Joseph <address@hidden> writes:

> > This number is sorted into the array to the other sources. The first 
> > item of the array is being connected maybe every 0.1 seconds. (or 
> > whatever this delay is set to). And the array has a max_length. If 
> > max_length is reached, the worst rated sources fall out. After a 
> > source was connected, it is rated again: If it has needed chunks, the 
> > rating is increased (rating=old_rating+2), if the connection fails, 
> > it is lowered (e.g. -3), if it has rare chunks it is increased even 
> > more (+4) and so on.
> 
> so the first source you find will have +4 ... even if it's the most 
> popular chunk, even once you have downloaded the chunk. seems to be 
> complex for the chunk availability. it would need a dynamic rating 
> (does the source still have the chunk i need, is the chunk still rare). 
> also, if a source get +4 for a rare file, it doesn't mean it will be 
> interesting for a second one
> nevertheless, the idea is interesting ...

Lets throw in my 2 c of thougths:

1. new source should be tried within a reasonable time limit but they
should not flood out older sources. Each file could get a next_new
counter. Every time a source is added the earliest time it might be
tried would be the next_new counter and the counter is increased. If
the next_new is too low it should be advanced to say a second from now
or something.

2. each file should have its own queue. Otherwise popular files would
kill sources of sparse files.

3. Each file should have a priority queue.
- The priority should be initialised with some small offset depending
  on the IP. Known dynamic IPs get a penalty.
- When a connection is made or failes the priority gets addjusted up
  or down.
- When the client is identified the client and version give some small
  adjustment to the priority. Nothing big but some.
- If a client has chunks I need it gets a bonus for each. _Otherwise_
  for each chunk it needs from me it gets a malus. (its more likely
  the client will get a chunk I already have than a new one).
  Sparseness of chunks could be considered here. 
- If I get some upload from a client the source should get a
  bonus. Maybe the speed could be factored in too.
- The time since the last upload and last connect should be factored
  in. A successful connect without a download isn't neccessarily a
  good thing.
- The queue should have a high and low watermark (as it has now). If
  the high mark is reached the worst sources are purged one by one
  until the low mark is reached. If that means puring new sources you
  never connected to so be it. One could just purge the worst purge
  whenever the queue is full but then it might allways be the newfound
  super fast but untred source. With a high/low mark system there is a
  certain time between finding more sources and purging where new
  connects can be tested.

Additionally I think the penalty for a connection failure for known
dynamic IPs should be very high. Its quite unlikely that a dialin user
has its donkey down for an hour and the start it up again. Its far
more likely he got a new IP.

4. All file queues should be in a priority queue. Its priority there
should depend on the top entry of each queue and the file itself. X
connects/second are then tried from that queue.
- each file should have a next_try counter.
- the priority of a file should be factored in. High priority files
  should be tried more often.
- the priority of the top element of each queue should be factored in

The easiest way would be to increase the next_try counter according to
the files priority and the priority of the top element of that queue
each time the queue is tried.

5. To prevent a source flooding a queue with only new sources between
the high and low watermark or in other words no new source has been
tried since the last purge of the queue could be locked to not accept
any new sources till all existing once have been tried. Sources that
get a malus on their first try would be removed from the queue, other
kept. When all new sources have been tried and the queue is still at
the high watermark the normal purging is done.

6. the next_new counter should probably be used for other sources
too. Every time a source disconnects it should get a next_retry_time
from the next_new counter if thats greater than the normal retry time
or the average of both. Alternatively two queues could be used. One
for existing sources and one for new ones and the two could be used
alternating (unless the retry time is not yet reached). Actually I
like that more.

7. a source that has multiple files should be added to the file with
the highest priority. This could be just the user set priority or
derived from the user set priority, the sparseness of the file/blocks,
the age of the file, the number of active connects and sources for the
file, ... The user set priority should probably be the main facor here
and the other factors should only make up for a bit, say +-10 points
on the user set priority. Setting a user set priority of 5 would mean
prever this but also get others. Setting it to 100 would mean
absolutely get this.

May the Source be with you.
                        Goswin