Re: [Vrs-development] Distributed Filesystem

[Chris Smith]

On Friday 12 July 2002 15:28, Ian Fung wrote:
> ok i hear you on the point that it is theoretically possible. the
> point about the unreliability being extended from the hardware to
> also include the network is a valid point. there is a difference
> though. in the case with hardware, failures are not expected too
> often and usually do not happen in mass quantities. for example in
> a RAID, its fine if one harddrive fails, but you wouldnt expect
> half the harddrives to. another thing is that LDS's are allowed and
> expected to come and leave as they see fit. this makes things way
> more complicated when trying to meet the requirements.

Exactly.
I really haven't got my head around how the propogation of data segments is 
going to work.  However I do feel strongly that it should be transparent to 
the application layer (the LDS itself) and handled by the RM layer directly.

[Which brings in a previous email of mine which stated that there is a little 
crossover between the RM and the CM as the CM needs to keep the RM updated as 
to what LDS's are currently active in the cluster (VRS).  This is just 
messaging in the end, the CM dispatches an event to the RM to say that 
something has changed and supplies it with enough info to sort out the 
Resource image.
I'm tackling everything from a messaging and abstraction point of view as I 
see it as the key to breaking the problem up into managable sub-projects].

> if you want to try to design some of the parts or even
> identify some of the issues to talk about when designing a
> distributed file system, then i think i can help a lot. i was
> recently involved in a project very similiar and have thought about
> a lot of the issues surrounding the design of a distributed file
> system. for example, you need to provide a way for a safe exit
> where a LDS will push the info it has to other nodes for
> redistribution. 

This should happen when an LDS joins the cluster so that it's data is 
immediately (or quickly) distributed across the cluster.  Any LDS in the 
cluster can then access the data.  (Some data/resources served by an LDS may 
be private and thus will not be shared - and so if that LDS goes away 
(failure/network outage etc) then those resources will be unavailable).

> ... also there needs to be schemes for recovery when a
> LDS fails. etcetc. i dunno where chris + bill stands in this
> aspect, but maybe we should do an irc meeting sometime.

If an LDS's (public) data/resourcdes are shared with the cluster on joining, 
then a failure will not be fatal as far as the VRS is concerned as there will 
be copies of the resoureces elsewhere.  The resource manager will take care 
of locating and retrieving the data.  The Cluster manager keeps the RM up to 
date as to what LDS's are available.
As the retrieval of data is done through messaging, Goldwater simplifies the 
whole thing by performing namespace and data dependant routing of request 
messages to a suitable LDS.


IRC at some point would be good.

-- 
Chris Smith
  Technical Architect - netFluid Technology Ltd.
  "Internet Technologies, Distributed Systems and Tuxedo Consultancy"
  E: address@hidden  W: http://www.nfluid.co.uk