Re: [Gluster-devel] How do we identify peers?

[Kaushal M]

On Thu, Jun 27, 2013 at 2:30 AM, Joe Julian <address@hidden> wrote:
> On 06/25/2013 11:37 PM, Kaushal M wrote:
>>
>> Forgot the link to the bug,
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=890502
>> Comment 2 in particular shows all the described problems
>>
>> On Wed, Jun 26, 2013 at 12:06 PM, Kaushal M <address@hidden> wrote:
>>>
>>> Hi all,
>>> We have a problem with how we identify peers in GlusterFS which we
>>> need to solve.
>>>
>>> Peers in GlusterFS are currently identified using two keys.
>>>   - the address of the peer (hostname/ip)
>>>   - a UUID
>>> The address is used as an identifier in the gluster cli commands and
>>> is the key visible to the users of glusterfs. The UUID is used
>>> internally by glusterd to identify peers.
>>> This approach works well when we are using glusterfs on systems with
>>> single network interfaces and when we are using a hostname as the
>>> address. We run into problems when we multiple network interfaces are
>>> involved or if hostname and ips are mixed in gluster cli commands.
>>> Bug-890502 [1] is a good example of the problems we have. This bug
>>> involves usage of multiple interfaces as well as mixing of
>>> hostname/ip.
>>>
>>> We need to come up with a good solution to this which would solve this.
>>>
>>> Some of the solutions that have been thought of are below.
>>>
>>> 1. Continue with the current implementation of using UUID internally
>>> and addresses externally, but provide a good translation interface
>>> which will translate a given address into UUID correctly. The
>>> translation interface might involve some network requests which could
>>> increase the time taken for command execution.
>>>
>>> 2. Use the UUID itself both internally and externally, and have a
>>> command for associating a list of addresses with a UUID. The downside
>>> is that typing out UUIDs is unwieldy and users will need users to
>>> execute more commands.
>>>
>>> 3. Similar to 2, but use an alias for the peer externally instead of
>>> the UUID. This will need an additional mechanism to specify the alias
>>> for a peer. This approach will need users one additional command
>>> compared to approach 2, but the other commands will be simpler to
>>> type.
>>>
>>>
>>>
>>> What are your thoughts about the problem and the proposed solutions?
>>> If anyone has any questions regarding the problem or the solutions, or
>>> if there is a better solution, please sound off.
>>>
> Here's my opinion on that from pre-acquisition:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=765437
>
> [FEAT] Use uuid in volume info file for servers instead of hostname or ip
> address
>
> In the volume info file, if the servers were referenced by uuid if the
> hostnames or ip addresses of the hosts change, the bricks would still
> reference the correct machine.
>

This is what we want to do, stop using addresses to identify peers.
We use UUIDs internally, but its usage is inconsistent. There are
places where we use UUID, as in the peer commands, and places where we
use addresses, as in the volume commands. Externally at least, we've
always used addresses, but it has drawbacks which we are trying to
fix.
We want to have a consistent way to identify peers everywhere. UUIDs
do this, but are a pain to use with the commands.
We could still continue using addresses with commands and have  a
robust translation layer which will translate any given address to a
UUID internally. Or we could have a short and unique alias for each
peer, which has a 1-1 mapping to UUID, which will be used with
commands. When using UUIDs directly or using aliases we need a way to
associate addresses with peers, which Justin's concept does.

I'm right now leaning towards a hybrid approach of using a translation
layer and using connection groups. We could use any of the addresses
associated with a peer, with the commands. Internally, we will be
maintaining a list of addresses associated with peers which will make
the internal translation easier. We will use UUIDs exclusively
internally, this includes the volfiles and other generated config
files.
I'll write a detailed description of this and send it here for comments soon.

- Kaushal

> In the IRC channel [on 2011-10-05], we had someone whose network had been
> renumbered. He had created his volumes using IP addresses instead of
> hostnames. He was able to peer probe which changed the host information to
> accurately reflect the new ip addresses, but the volumes still pointed at
> the old addresses. They could not run replace-brick to update the addresses
> of their bricks.
>
> If the bricks were referenced by host uuid, the peer probe would have
> allowed the bricks to resolve to the new information without further
> changes.