[Gluster-devel] Help developing new translator

[michael . osullivan]

Hi Gluster developers,

I'm would like to develop a new translator so I can experiment with
different methods of spreading data around a Gluster system, but I have no
Gluster experience and very little filesystem development experience. I
have tentatively called the new translator "spread" and attach my initial
thoughts for functionality below as a starting point for discussion. Any
help/advice/suggestions/feedback you could give me would be very welcome.
If I could better understand the basics behind translators I may be able
to cobble together what I need from stripe, distribute and replicate.

Kind regards, Mike O'Sullivan

WRITE

1) File arrives, goes to "generate" function and gets key + spread version
number;
2) File split into smaller pieces (cf. stripe) and "place" function uses
the key and spread version number to assign a replication sequence = set
of Gluster bricks to each piece. Note. Pieces may be assigned the same
replication sequence;
3) Each piece is replicated to all bricks in the replication sequence,
note that some bricks may appear in more than one sequence.

Example.

File size = 50MB
Atom size = 10MB

---------------------
| 1 | 2 | 3 | 4 | 5 |
---------------------

1 goes to replication sequence (Brick 1, Brick 2)
2 goes to replication sequence (Brick 1, Brick 3)
3 goes to replication sequence (Brick 1, Brick 2)
4 goes to replication sequence (Brick 2, Brick 3)
5 goes to replication sequence (Brick 1, Brick 3)

Brick 1

---------------------
| 1 | 2 | 3 | - | 5 |
---------------------

Brick 2

---------------------
| 1 | - | 3 | 4 | - |
---------------------

Brick 3

---------------------
| - | 2 | - | 4 | 5 |
---------------------


READ

1) Request for file arrives and key and spread version number get read
from metadata (using extended attributes?);
2) The "locate" function uses the key and spread version to recover the
replication sequence used to write each piece of the file;
3) The piece is read from the first brick in the sequence if possible
(brick may have failed or be unreachable), otherwise it is read from the
next brick in the sequence, etc until either the piece is read or all
bricks in the replication sequence have failed.

LS, MKDIR, etc
I'm not sure what would be needed for these operations.

SELF-HEAL
A file should be able to heal itself using the replicas of pieces, but I'm
not sure how this should be done.

NOTE. The functions generate, place and locate can be implemented to work
for any size piece and any number of replications. I would also like to
use an UPDATE to dynamically update the data used to drive these
functions, hence the need for spread version number.