Re: [Vrs-development] Scalability

[Ian Fung]

> Chris, what do you know about how to write middleware that supports 
> millions of nodes? :-)
> 
> I'll certainly listen to ideas...

well. what about something like this. we can treat all message routing
on demand. by message, i dont mean every packet. let's treat it as a
truly decentralized p2p network. for example, discovery of services
would be done dynamically. one thing to think about is why we want every
LDS to keep a list of all other LDS's. things that pop into mind are:
routing and list of services (chris: if there are more, please let me
know).  if we can do all those things dynamically, there will be no need
for that static list. one thing that i can forsee it complicating a lot
is doing sychronization and consistency of the distributed filesystem.
we would need very intelligent algorithms in the CM to deal with this.
this is the only thing that bugs me. everything else in VRS can pretty
much be done without coordination between all the nodes. if i need a
webservice, i can dynamically find an LDS node with and just communicate
with that node. or if i need computing power or whatever, i can send out
requests for the work and wait to see who responds. they dont have to be
done by specific nodes. this alternative is not all bad. good things
that will emerge from this if it is designed correctly is that adhoc
participation of nodes will not be an issue. we can also run VRS in an
adhoc multihop environment.

this is of course a radical change in philosophy from what Goldwater
assumes about the underlying network topology. if there is a way to add
functionality to Goldwater to support all the things i mentioned
earlier, we should design it such that if we wanted, we can plug n play
those components. for example, just make discovery of nodes dynamical
and everything else will use a static LDS list (well the list wont be
static but it'll be an entire list of all LDS's in the VRS). these are
some of the things we need to talk about.

Chris and I are meeting probably saturday to talk about macs's place in
VRS. i am also going to talk about VRS at this sunday's auth meeting.
during this saturday's meeting, i would also love to have vrs in a state
where actual coding can begin to happen. we need to move a level lower
than what is on the vrs website. that way, we can also make decisions
about other things that depend on implementation, such as macs. also, i
would like to sketch out a security model for vrs. so Eric, let me know
if you can make it this weekend. i want all three of us to meet about
this.

-alias