Re: [GNUnet-developers] DistribNet and GNUNet

[Kevin Atkinson]

On Tue, 23 Apr 2002, Christian Grothoff wrote:

Thank you for your feedback.

> On Tuesday 23 April 2002 06:40 am, you wrote:
> > I am starting a new distributed network, DistribNet, which has similar
> > aims of Freenet and GNUNet except that my main focus is speed and
> > stability rather than anonymously.  Compared to freenet I like your
> > network a lot better.  If for now other reason is that it is not written
> > in god dam Java.  Sorry about that, it is just that I hate Java in just
> > about every way possible.  Seriously though, I like most things about your
> > project except for your extreme approach of using UDP packets for
> > *everything*, tiny 1K block sizes, and using the filesystem to store these
> > tiny blocks.  I plan to use a mixture of UDP and TCP packets and 32K block
> > sizes for splitting up files.
> 
> I have a couple of remarks. 
> a) we are *currently* using UDP, but we are planning to support TCP 
>    in the future (end of summer?).

Will you be using persistent connections?

> b) what is the advantage of using 32k blocks? If you need to transfer 
>    a 100 MB file, you need to transfer 100 MB. The number of datagrams
>    send is the same because your 32k blocks will be fragmented into
>    approximately 1k fragments anyway. The total amount transferred will
>    always be 100 MB + a few percent overhead, but does it really matter
>    if it is 104 or 105 MB? I doubt it. 

For one thing I am probably not using these blocks the same way you are.  
Everything will not be exactly 32K.  The 32K blocks will be used for 
transferring files and to allow clients to only transfer part of a file or 
get files from multiple hosts.  Other communications will be done in what 
ever block size is most suitable for the message.

Also, the more blocks the more queries and the more work the nodes have to 
do.

> c) In the case of small files, you have to do more padding and will be
>    less efficient than GNUnet.

I won't be using fixed size blocks.

> 
> > Below is an outline of DistribNet.  Let me know what you think.
> >
> > Perhaps will can work together provided our design goals don't conflict
> > too much.  Maybe in the future we can even merge the two networks.
> 
> That's premature since you're still drafting your basics. If you want to work 
> on improveing GNUnet by contributing code, you are welcome. I am open to 
> suggestions, but the bigger the change, the better the argument for it must 
> be.

Tell me if you had to chose between anonymity or speed which will you 
chose?  I will chose speed?  If you will chose anonymity I think our goals 
are incompatible.

> > I am most interested in your lookup services and accounting of GNUNet.
> 
> Well, read the research papers to get started, that's probably the best 
> documentation we have at the moment...

Ok. I will.

> > Feedback more than welcome.
> >
> >                               DistribNet
> >
> > A global peer-to-peer internet file system in which anyone can tap into
> > or add content to.
> >
> > Kevin Atkinson (kevin at atkinson dhs org)
> > Last Modified: 2002-04-22
> > Project Page: http://distribnet.sourceforge.net/
> > Mailing list: http://lists.sourceforge.net/lists/listinfo/distribnet-devel
> >
> > Meta Goals:
> >
> > *) To allow anyone, possibly anonymously, to publish web sites with
> >     out having to pay to for the bandwidth for a commercial provider
> >     or having to put up with the increasingly add ridden free web
> >     sites.  One should not have to worry about bandwidth
> >     considerations at all.
> 
> You will always have to worry about bandwidth. 

I really need to rephrase that.  I am talking about who is responsible 
for providing the bandwidth.  For a popular web site it will be the owner 
of the web site, with a distributed network it will be shared by all those 
who use the network.

> 
> > *) Bring back the sense of community on the Internet that was once
> >     present before the internet become so commercialized.
> >
> > *) Serve as an efficient replacement for current file sharing networks
> >     such as Morpheus and Gnutella.
> 
> Why? For certain applications, they seem to do the job. 

But they are problems with it that I would like to solve.

> That's like saying 
> freenet should try to replace the WWW. As users that are willing to install 
> spyware on their machines are hardly going for a secure network like GNUnet, 
> there will probably always be a number of users where gnutella fits the bill. 

But gnutella is horribly inefficient.

> If you want to build something, don't go for a userbase with millions of 
> happy users, go for a userbase with lots of unhappy users or where there is 
> nothing so far (just a suggestion). 

And that will be.

> 
> > *) To have the network stable and working before some Commercial
> >     company designs a propitiatory network similar to what I envision
> >     that can only be accesses via freely available but not FSF
> >     approved free license.
> >
> > (Possibly Impossible) Goals:
> >
> > *) *Really* fast lookup to find data.  The worst case should be O(log(n))
> >     and the average case should be O(1) or very close to it.
> 
> You are already looking at Chord (Mircosoft research), just be aware of the 
> drawbacks:
> * forced content migration
> * hosts fully exposed
> * how about dynamic networks (hosts join and levae quickly?)

So do you have a better solution?  I have looked at the Chord paper and 
the results look very promising for what I have in mind (which I admit I 
haven't completly described).

Also where did you get the idea that the Chord project is part of 
Mircosoft research?  I can find no such thing at there web site 
http://www.pdos.lcs.mit.edu/chord/.

> > *) Actually retrieving the data should also be really fast.  Popular
> >     data should be sitting on the same subnet.  On average it should
> >     be as fast or faster than a typical web site (such as slashdot,
> >     google, etc.).  It should make effective use of the
> >     topology of the internet to to minimize network load and maximize
> >     performance.
> 
> This would require the network to know the topology in the first place. Code 
> to do that discovery would be great for GNUnet, too.

Yes it would.  I have not figured out this part of the equation but I 
think it can be done.

> > *) General searching based on keywords will be build into the protocol
> >     from the beginning.  The searching faculty will be designed in
> >     such a way to make message boards trivial to implement.
> >
> > *) Ability to update data while keeping old revisions around so data never
> >     disappears until it is truly unwanted.  No one person will have
> >     the power to delete data once it spreads throughout the network.
> >
> > *) Will try very hard to keep all but the most unpopular content from
> >     falling off the network.  Basically before deleting a locally
> >     unpopular key it will first check if other nodes are storing the
> >     key and how popular they find the key.  If not enough nodes are
> >     storing the key and there is any indication that the data may be
> >     useful at a latter date it will not delete it unless it absolutely
> >     has to.  And if it does delete it it will first try uploading it
> >     to other nodes with more disk space available.
> >
> > *) Ability to store data indefinitely if someone is willing to provide
> >     the space for it (and being able to find that data in log(n)
> >     time).
> >
> > *) Extremely robust so that the only way to kill the network is to
> >     disable almost all of the nodes.  The network should still
> >     function even if say 90% of it goes down.
> >
> > *) Extremely effect cpu-wise so that a fully functional node can run in
> >     the background and only take 1-2% of the CPU.
> 
> Well, you want everything, but don't quite describe how to do it. Goals are 
> one thing, but the key is to have ideas how to get there. I don't see any 
> here.

I did say Possibly imposable goals, didn't I?

> > Applications:
> >
> > I would like the protocol to be able to effectually support (ie with out
> > any ugly hacks that many of the application for Freenet use)
> >
> > 1) Efficient Web like sites (with HTTP gateway to make browsing easy)
> > 2) Efficient sharing of files large and small.
> > 3) Public message forms (with IMAP gateway to make reading easy)
> > 4) Private Email (with the message encrypted so only the intended
> >     recipient can read it, again with IMAP gateway)
> > 5) Streaming Media
> > 6) Online Chat (with possible IRC or similar gateway)
> >
> > Anti-Goals:
> >
> > (Also see philosophy for why I don't find these issues that important)
> >
> > *) Complete anonymity for the browser.  I want to focus first on
> >     performance than on anonymity.  In fact I plan to use extensive
> >     logging in the development versions so that I track network
> >     performance and quickly cache performance bugs.  As DistribNet
> >     stabilizes anonymity will be improved at the expense of logging.
> >
> >     The initial version will only use cryptology when absolutely
> >     necessary (for example key signing).  Most communications will be
> >     done in the clear.  After DistribNet stabilizes encryption will
> >     slowly be added.  When I add encryption I will carefully monitor
> >     the effect it has on CPU load and if proves to be expensive I will
> >     allow it to be optional.
> >
> >     Please note that I still wish to allow for anonymous posting of
> >     content.  However, without encryption, it probably won't be as
> >     anonymous as Freenet or your GNUNet.
> 
> Well, without encryption you won't get accounting either. Without crypto you 
> will also not get integrity. Besides that, the encryption is not that 
> expensive. Read our paper 'anonymity for free', or why we believe that 
> anonymity can be obtained very cheaply.

I will look at your papers.  However, I plan to use encryption when 
necessary.  Just not for everything.

> 
> > *) Data in the cache will be stored in a straight forward manner.  No
> >     attempt will be made to prevent the node operate from knowing what
> >     is in his own cache.  Also, by default, very little attempt will
> >     be made to prevent others from knowing what is a particular node
> >     cache.
> 
> This would make the nodes vulnerable to attacks. But it's a design choice. In 
> GNUnet, you can have both, knowledge and no knowledge, user's choice. You may 
> want to consider that possibility.
> 
> > Philosophy:
> >
> > *) I have nothing against complete anonymity, it is just that I am
> >     afraid that both Freenet and GnuNet or more designed around the
> >     anonymity and privacy issues then they are around the performance
> >     and scalability issues.
> 
> Well, the performance and scalability issues in P2P are very hard, depending 
> on which scale you are shooting for; if you can solve them, great. I don't 
> think that the GNUnet design makes these problems harder or easier.

Do you think you will be able to make GNUNet as fast or faster than the 
Web?

---
http://kevin.atkinson.dhs.org