Re: [Gluster-devel] Choice of Translator question

[Gareth Bult]

>The trusted.afr.version extended attribute tracks while file version is 
>being used, and on a read, all participating AFR members should respond 
>with this information, and any older/obsoleted file versions are 
>replaced by a newer copy from one of the valid AFR members (this is 
>self-heal)

Yes, understood.

>I think they are planning striped reads per block (maybe definable) at a later 
>date.

Mmm, so at the moment, when it says AFR does striped reads, what it really 
means is that it does striped reads, just so long as you have lots of 
relatively small files and not a few large files .. ???

>Read from the the file from a client (head -c1 FILE >/dev/null to force)

OR find /mountedfs -exec head -c1 > /dev/null {} \;

.. which is good, but VERY inefficient for a large file-system.

>you could use the stripe translator over AFR to AFR chunks of the DB 
>file, thus allowing per chunk self-heal.

Mmm, my experimentation indicates that this does not happen. I've just spent 3 
hours trying to prove / disprove this with various configurations - AFR 
self-heals on a file basis, not on a stripe-chunk basis.

If I have 4 bricks, two stripes using 2 bricks each, then an AFR on top - any 
sort of self-heal replicates the entire DB.
If I have 4 bricks, two AFR's and one stripe on top, I get the same thing.


>I'm not familiar enough with database file writing practices in general (not 
>to mention your 
>particular database's practices), or the stripe translator to tell 
>whether any of the following will cause you problems, but they are worth 
>looking into:

We're talking about flat files here, some with append, some with seek/write 
updates.

>1) Will the overhead the stripe translator introduces with a very large file 
>and relatively small chunks cause performance problems? (5G in 1MB stripes = 
>5000 parts...)

No, this would be fine if the AFR/Stripe combination actually did a per-chunk 
self heal.

>2) How will GlusterFS handle a write to a stripe that is currently 
>self-healing?  Block?

The stripe replicates the entire stripe (which is big) and both read and write 
operations block during the heal.

>3) Does the way the DB writes the DB file cause massive updates throughout the 
>file, or does it generally just append and update the indices, or something 
>completely different.  It could have an affect on how well something like this 
>works.

I don't think access speed is an issue, glusterfs is very quick. The issue is 
recovery, it appears not to operate as advertised!

>Essentially, using this layout, you are keeping track of which stripes have 
>changed and only have to sync those particular ones on self-heal. The longer 
>the downtime, the longer self-heal will take, but you can mitigate that 
>problem with a rsync  of the stripes between the active and failed GlusterFS 
>nodes BEFORE starting glusterfsd onthe failed node (make sure to get the 
>extended attributes too).

Ok, firstly, manual rsync's sort of defeat the object of the exercise.
Secondly, having to go through this process every time a configuration is 
changed / glusterfsd is restarted is unworkable.
Thirdly, replicating many GB's of data hammers the IO system and slows down the 
entire cluster - again undesirable.

Being able to restart a glusterfsd without breaking the replica's would help, 
but I see no mention of this ...

>The above setup, if feasible, would mitigate restart cost, to the point where 
>only a few megs might need to be synced on a glusterfs restart.

Ok, well I appear to have both AFR and Striping working and I can observe their 
operation at brick level and confirm they are working Ok.

Here's my basic test harness;

On the client system;

$dd if=/dev/zero of=/mnt/stripe/database bs=1M count=1024

write.py
#!/usr/bin/python
io=open("/mnt/stripe/database","r+")
io.seek(1024*1024*900)
io.write("Change set version # 6\n")
io.close()

On the bricks I have;

read.py
#!/usr/bin/python
io=open("/export/stripe-1/database","r+")
io.seek(1024*1024*900)
print io.readline()
io.close()

When I run write.py on the client, both bricks show the correct change.
Then I kill glusterfsd on brick2.
Running write.py on the client shows an update on brick1, obviously not on 
brick2.
Restarting glusterfsd on brick2 shows a reconnect in the logs.
On the client; head -c1 database
Initiates a self heal, shown in the logs with DEBUG turned on
Running read.py on brick1 and brick2 blocks ...
An entire 1G chunk is copied to brick 2
read.py on bricks 1 and 2 then continue when the copy finishes ..

(!)

I'm using fuse-2.7.2 from the repos and gluster 1.3.7 from the stable tgz ...

fyi; The fuse that comes with Ubuntu/Gutsy seems to cause gluster to crash 
under write-load, I'm still waiting to see if the current CVS version solves 
the problem ...

Gareth.