Re: [Gluster-devel] [Gluster-users] Phasing out replace-brick for data migration in favor of remove-brick.

[Anand Avati]

On Fri, Sep 27, 2013 at 1:56 AM, James <address@hidden> wrote:

On Fri, 2013-09-27 at 00:35 -0700, Anand Avati wrote:
> Hello all,
Hey,

Interesting timing for this post...
I've actually started working on automatic brick addition/removal. (I'm
planning to add this to puppet-gluster of course.) I was hoping you
could help out with the algorithm. I think it's a bit different if
there's no replace-brick command as you are proposing.

Here's the problem:
Given a logically optimal initial volume:

volA: rep=2; h1:/b1 h2:/b1 h3:/b1 h4:/b1 h1:/b2 h2:/b2 h3:/b2 h4:/b2

suppose I know that I want to add/remove bricks such that my new volume
(if I had created it new) looks like:

volB: rep=2; h1:/b1 h3:/b1 h4:/b1 h5:/b1 h6:/b1 h1:/b2 h3:/b2 h4:/b2
h5:/b2 h6:/b2

What is the optimal algorithm for determining the correct sequence of
transforms that are needed to accomplish this task. Obviously there are
some simpler corner cases, but I'd like to solve the general case.

The transforms are obviously things like running the add-brick {...} and
remove-brick {...} commands.

Obviously we have to take into account that it's better to add bricks
and rebalance before we remove bricks and risk the file system if a
replica is missing. The algorithm should work for any replica N. We want
to make sure the new layout makes sense to replicate the data on
different servers. In many cases, this will require creating a circular
"chain" of bricks as illustrated in the bottom of this image:
http://joejulian.name/media/uploads/images/replica_expansion.png
for example. I'd like to optimize for safety first, and then time, I
imagine.

Many thanks in advance.

I see what you are asking. First of all, when running a 2-replica volume you almost pretty much always want to have an even number of servers, and add servers in even numbers. Ideally the two "sides" of the replicas should be placed in separate failures zones - separate racks with separate power supplies or separate AZs in the cloud. Having an odd number of servers with an 2 replicas is a very "odd" configuration. In all these years I am yet to come across a customer who has a production cluster with 2 replicas and an odd number of servers. And setting up replicas in such a chained manner makes it hard to reason about availability, especially when you are trying recover from a disaster. Having clear and separate "pairs" is definitely what is recommended.

That being said, nothing prevents one from setting up a chain like above as long as you are comfortable with the complexity of the configuration. And phasing out replace-brick in favor of add-brick/remove-brick does not make the above configuration impossible either. Let's say you have a chained configuration of N servers, with pairs formed between every:

h(i):/b1 h((i+1) % N):/b2 | i := 0 -> N-1

Now you add N+1th server.

Using replace-brick, you have been doing thus far:

1. add-brick hN:/b1 h0:/b2a # because h0:/b2 was "part of a previous brick"

2. replace-brick h0:/b2 hN:/b2 start ... commit

In case you are doing an add-brick/remove-brick approach, you would now instead do:

1. add-brick h(N-1):/b1a hN:/b2

2. add-brick hN:/b1 h0:/b2a

3. remove-brick h(N-1):/b1 h0:/b2 start ... commit

You will not be left with only 1 copy of a file at any point in the process, and achieve the same "end result" as you were with replace-brick. As mentioned before, I once again request you to consider if you really want to deal with the configuration complexity of having chained replication, instead of just adding servers in pairs.

Please ask if there are any more questions or concerns.

Avati

 

James

Some comments below, although I'm a bit tired so I hope I said it all
right.

> DHT's remove-brick + rebalance has been enhanced in the last couple of
> releases to be quite sophisticated. It can handle graceful decommissioning
> of bricks, including open file descriptors and hard links.

Sweet

>
> This in a way is a feature overlap with replace-brick's data migration
> functionality. Replace-brick's data migration is currently also used for
> planned decommissioning of a brick.
>
> Reasons to remove replace-brick (or why remove-brick is better):
>
> - There are two methods of moving data. It is confusing for the users and
> hard for developers to maintain.
>
> - If server being replaced is a member of a replica set, neither
> remove-brick nor replace-brick data migration is necessary, because
> self-healing itself will recreate the data (replace-brick actually uses
> self-heal internally)
>
> - In a non-replicated config if a server is getting replaced by a new one,
> add-brick <new> + remove-brick <old> "start" achieves the same goal as
> replace-brick <old> <new> "start".
>
> - In a non-replicated config, <replace-brick> is NOT glitch free
> (applications witness ENOTCONN if they are accessing data) whereas
> add-brick <new> + remove-brick <old> is completely transparent.
>
> - Replace brick strictly requires a server with enough free space to hold
> the data of the old brick, whereas remove-brick will evenly spread out the
> data of the bring being removed amongst the remaining servers.

Can you talk more about the replica = N case (where N is 2 or 3?)
With remove brick, add brick you will need add/remove N (replica count)
bricks at a time, right? With replace brick, you could just swap out
one, right? Isn't that a missing feature if you remove replace brick?

>
> - Replace-brick code is complex and messy (the real reason :p).
>
> - No clear reason why replace-brick's data migration is better in any way
> to remove-brick's data migration.
>
> I plan to send out patches to remove all traces of replace-brick data
> migration code by 3.5 branch time.
>
> NOTE that replace-brick command itself will still exist, and you can
> replace on server with another in case a server dies. It is only the data
> migration functionality being phased out.
>
> Please do ask any questions / raise concerns at this stage :)

I heard with 3.4 you can somehow change the replica count when adding
new bricks... What's the full story here please?

Thanks!
James

>
> Avati
> _______________________________________________
> Gluster-users mailing list
> address@hidden
> http://supercolony.gluster.org/mailman/listinfo/gluster-users