----- Original Message -----
Hi,
As of today most gluster commands take a cluster wide lock, before performing
their respective operations. As a result
any two gluster commands, which have no interdependency with each other,
can't be executed simultaneously. To
remove this interdependency we propose to replace this cluster wide lock with
a volume specific lock, so that two
operations on two different volumes can be performed simultaneously.
By implementing this volume wide lock, our agenda is to achieve the
following:
1. Allowing simultaneous volume operations on different volumes. Performing
two operations simultaneously on the same
volume should not be allowed.
2. While a volume is being created, or deleted, operations(like rebalance,
geo-rep) should be permitted on other volumes.
You can't meaningfully interleave a volume-delete and a volume-rebalance on the
same volume.
The locking domains must be arranged such that, create/delete operations
take a lock which conflict all locks held on any volume. This would
handle the scenario mentioned above.
The volume operations performed in the cluster can be classified into the
following
categories,
- Create/Delete - Need a cluster-wide lock. This locks 'trumps' all other locks
- Add-brick, remove-brick, replace-brick, rebalance, volume-set/reset and a
whole bunch of features
associated with a volume like quota, geo-rep etc - Need a volume-level lock.
- Volume-info, volume-status - Need no locking.
3. Two simultaneous volume create or volume delete operations should not be
permitted.
We propose to do this in two steps:
1. Implementing the volume wide lock: In order to implement this, we will add
a lock consisting of the uuid of the
originator, to the in-memory volinfo(of that particular volume), in all
the nodes of the cluster. Once this lock is
taken, any other command for the same volume, will not be able to acquire
this lock from that particular volume's
volinfo. Meanwhile other operations on other volumes can still be
executed.
There is one caveat with storing the volume-level lock in volinfo object.
All the nodes are not guaranteed to have an up-to-date version of the volinfo
object. We don't know have the necessary mechanism to select the peers based
on their recency of volume information. Worse still, the volinfo object could be
modified on incoming updates from other peers in the cluster,
if and when they rejoin (the available partition of) the cluster, in the middle
of
a transaction. I agree this lack of guarantee is part of a different problem,
but this is a runtime reality :(
This prompts me to think that we should keep all the lock related book-keeping
independent of things that are not guaranteed to stick around, for the lifetime
of a command. This way we can keep the policy of locking (ie. volume-wide Vs
cluster-wide)
separate from the mechanism.
cheers,
krish
2. Stop using the cluster lock for existing commands: Port existing commands
to use this framework. We will use
op-version to take care of backward compatibility for the existing
commands. We need to take care of commands like
volume create, volume delete, rebalance callbacks, implicit volume syncs
(when a node comes up), the volume sync
command which modify the priv->volumes, and also other non-volume
operations which work inside the gambit of the
cluster locks today while implementing this.
Please feel free to provide feedback.
Regards,
Avra
_______________________________________________
Gluster-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/gluster-devel