Re: [Gluster-devel] Can I bring a development idea to Dev's attention?

[Ed W]

 On 24/09/2010 08:25, Shehjar Tikoo wrote:
> Thanks, we have similar locking improvements in mind but cannot 
> promise a date when these will be available. Some of the challenges 
> that we'll need to think about are how to map any such locking scheme 
> to standard locking behaviour for posix/nfsv3/v4/cifs.

Hi, thanks for replying

Whilst I can see that there is some optimisation to be had by combining 
brick level locking with filesystem level locking, I just want to 
clarify that my proposal was really about intra-brick locks, and not 
really about the toplevel filesystem level locking?

Just to clarify (and apologies if I'm trying to teach filesystem experts 
really obvious stuff...)

- The goal of any fileserver is to take async requests from lots of 
clients and arrange to serialise that access
- Up until recently such fileservers have been on a single machine, but 
involving multiple async clients connecting
- Even in a single server solution the bottleneck becomes that each 
client cannot cache *any* data since it's not known if the server copy 
has changed since we accessed it (even a microsecond earlier)
- The solution which has become popular (see CIFS, NFSV4 (?), GFS2, etc) 
was to offer clients an "optimistic lock", ie the client can acquire a 
token which while it's held means that it can cache data locked by that 
token and even offer writeback optimisations on that data (obviously 
subject to whatever the application tolerates for unsync'ed data)
- This "optimistic lock" means that we effectively push the file locking 
to the client, hence once a lock is acquired then further access by the 
client is no longer bounded by the network access latency, under many 
circumstances this leads to massive speedups
- Clearly when a second client comes along and demands access to the 
same data then we need a process to break the lock and inform the first 
client that they need to reacquire the lock (or revert to a kind of 
"write-through" access system while waiting)

So this process clearly benefits situations where there is serialised 
access by single clients at a time.  Excluding databases however, this 
access pattern seems quite common for lots of applications

So with regards to Gluster I would see that we need this same type of 
locking implemented at the brick level.  Hence if you re-read the 
description above, then each *gluster server* would be the possible 
clients (think of the lower level being bricks talking to each other, 
and the upper level being clients talking to bricks).  ie yes, posix 
locking needs to serialise access to every end client that connects to 
every brick, but we can also benefit from locking to serialise access 
between bricks (if 3,000 clients hammer one brick for a single file, 
then we care that our single brick is allowed to read/write that file 
freely because it informed the other bricks that it now holds a lock, 
it's a separate problem to serialise all the clients talking to the one 
brick)

So compared with traditional fileservers we actually need two levels of 
locking to serialise access.  At one level we need to serialise clients 
access to the filesystem, and lower down we need to serialise access 
between bricks

I think an alternative way of looking (and perhaps implementing) the 
situation could be something like:

- Consider two bricks with files replicated between them
- Client 1 accesses Brick 1 and requests File A
- Brick 1 contacts the other replicas and requests to become the "master 
replica" of that file. All future accesses to that file must now go 
through only Brick 1 while it remains in that "role"
- If Client 2 accesses Brick 1 and tries to do something with File A, 
then the normal filesystem locking must arrange for serialisation 
between Client 1 and Client 2, however, Brick 1 need not contact any 
other brick and there is no network latency penalty serving that file to 
Client 2 (obviously at some point one client will write data and we need 
to sync that, but read access incurs no network access)

- OK, now the trick is what happens when Client 3 accesses Brick 2 and 
requests File A...  Somehow we need to wrest control back from Brick 1 
and inform it that it's no longer the "master".  A really simple 
solution to this (at least conceptually) is to proxy all access requests 
from Brick 2 back to Brick 1.  This satisfies our requirement that 
accesses are serialised across bricks and effectively there is still a 
"master" brick remaining in control.
- We can see that this setup is conceptually similar to having a 
traditional lock server arbitrating brick access to a given file, but in 
example above we have implemented a distributed lock server, the lock 
server effectively becoming the same server as what we hope is the "hot 
server", so that we aren't incuring network latency to contact the lock 
server all the time.
- A further improvement would clearly be to have some kind of process 
where the "master brick" can move about, ie in the case above if Client 
3 starts to bash away at Brick 2 for File A, then Brick 2 is migrated to 
become the "master" and hold the lock, now any access through Brick 1 
must effectively proxy requests back to Brick 2 or re-acquire it's lock 
(ie become the master)

OK, so the above is a very simple example of optimistic locking and 
could be trivially implemented using an external lock server which 
tracks which brick currently holds the lock for a given file (ie can 
read/write freely without first checking if other bricks have modified 
the file).  A given brick which doesn't hold a lock on a file must first 
do kind of what it does already and contact the lock server to see if 
another brick holds the lock.  If not it can acquire the lock itself.  
If the lock is held elsewhere we either need to break the lock (or proxy 
access requests to the server holding the lock).

Really this is not so different to what is there today, but it's simply 
an efficiency improvement because we don't need to touch *every* brick 
for *every* file access, instead we make some network requests on first 
access to a file and then can continue to touch that file for a period 
afterwards without needing further network access with other bricks

However, whilst some kind of implementation of the above could offer a 
huge performance speedup for many of the situations which come up on the 
mailing list, the issue is that the lock server becomes a) a bottleneck 
and b) point of failure.  So the chain of thought almost certainly goes 
something like:

- Make the gluster bricks become the lock servers, ie they negotiate 
amongst themselves. Really this is roughly what happens right now, only 
it's on every access, rather than access being "sticky" once acquired
- Now analyse all the corner cases that bricks go down holding locks, or 
get segmented while holding/acquiring locks and discover some tricky 
issues...

Paxos seems like a clever way of dealing with the locking going 
distributed, yet not necessarily having a 100% consistent view of who 
owns which lock.  By introducing a voting method it can show robustness 
in the face of failed machines and new machines can be added without 
needing to store reliable state information (or at least this is true 
with the improvements described in the articles)



Does that make sense?  Apologies if the above is long winded, but the 
point is really that the performance improvements come from pushing 
locks between bricks, and probably this is distinct from client level 
locking such as nfs/cifs/posix, etc locking

For advanced cluster filesystems such as GFS2, the general "optimistic 
locking" technique appears to show massive speed improvements (for many 
access patterns) and it's also likely to do so in Gluster.  Really my 
original email jumped two steps and suggested an improved form of 
distributed locking, which itself could be used as the actual 
implementation, but other forms of distributed locking between bricks 
would be highly desirable also.

Thanks for listening

Ed W