Re: [Gluster-devel] Client side AFR race conditions?

[Derek Price]

address@hidden wrote:
> On Wed, 7 May 2008, Anand Avati wrote:
>
> > > The only way I see to ensure data integrity is to have some arbiter vet
> > > > all writes.  You can try to make that arbiter redundant, but good luck
> > > > making it actually distributed.
> > >
> > > I've seen the distributed arbiter done in proprietary software, so it
> > > must be possible.  The design is pretty clear to me, but I have no idea
> > > where to start integrating the idea into glusterfs, though gluster's the
> > > closest thing to what I need that I've seen in open source.
> >
> > Can you give some details/links? We would be interested to learn about 
> > it.
>
> I suspect what was referred to was a system where the locks are notified 
> to every host, not an actually load sharing system. DLM (RHCS/GFS) does 
> it by multicasting, presumably with acknowledgements being returned from 
> each connected node. I've not looked at the DLM protocol in great 
> detail, so I don't know what the details are.

Actually, I was thinking of WANdisco's Multi-site CVS/SVN/MySQL 
mirroring software.  It's not generalized to the point of being a disk 
load sharing system, exactly, but I think the concept and the problems 
are the same.  They use a quorum locking model and basically journal the 
transaction with whichever server they are wrapping for later replay on 
the other servers.

There used to be a white-paper on WANdisco's protocol online (I haven't 
looked recently).  I didn't know much about DLM (and, after reading what 
documentation I could find online just now, I don't feel like I know 
much more), but it sounds like DLM uses a similar quorum model for locking.

As for the versioning (and perhaps this is relevant to the discussion 
taking place in another thread), I don't see how this can be done 
without meta-data journaling, so why not make things even simpler and 
share a unique version number between all entities changed in a 
transaction?  So, for any server to acquire an implicit write lock, the 
quorum must agree to increment a global transaction ID (which could also 
be attached in the FS as a directory and/or file's version number). 
Then, as long as any given system knew that its journal/replay was 
up-to-date with the latest transaction ID according to the quorum, then 
it could trust a file's content without consulting a file-specific 
revision number.

If a server was not completely up-to-date, then it would at least have 
to synchronize the meta-data journal and consult it to find if a 
requested file had any pending writes and decide whether it needed to 
synchronize the file before serving it.

Regards,

Derek
--
Derek R. Price
Solutions Architect
Ximbiot, LLC <http://ximbiot.com>
Get CVS and Subversion Support from Ximbiot!

v: +1 248.835.1260
f: +1 248.246.1176