Re: [Gluster-devel] glusterfs_1.4.0qa19: small issues

[Brent A Nelson]

On Tue, 10 Jun 2008, Brent A Nelson wrote:

> Okay, that's fixed, along with the removexattr issue.  Another couple of 
> quirks; cp -a /usr/bin /gluster/bin makes a perfect copy, except:
>
> 1) /gluster/bin itself doesn't preserve the correct permissions (rsync 
> succeeds, though, and a cp -a to /tmp also works fine, so it's not a problem 
> with cp); the contents are all fine, except:
>
> 2) /gluster/bin/sudoedit did not have the setuid bit set (this fails with 
> rsync, too, but a manual chmod u+s succeeds).

I tortured my GlusterFS pretty heavily today.  Apart from the issues 
above, it held up perfectly for repeated rsync/rm cycles.

I noticed that when doing a large (10GB ) dd write on a client, if the 
write was going to the client node (which is also a server), ls would hang 
for long periods on that client.  So, I figured it was time to load up 
io-threads (just 2 threads per export) on the servers to split up the 
metadata and io traffic.  It worked beautifully; ls -al was very 
responsive on all clients, even with heavy write activity.

However, I noticed that with io-threads loaded, the server processes would 
sometimes consume several GB of RAM (up to 2.2GB).  I didn't worry about 
it too much, as it generally would free the memory back.  I did more 
exhaustive testing, with all 4 servers acting as clients, all 4 doing dd 
writes, and it worked fine for awhile.  Eventually, however, one of my 
nodes ran out of memory; perhaps that extra io-threads memory consumption 
is an issue after all.

Also, when the node ran out of memory, my other GlusterFS clients hung. 
As someone mentioned previously, GlusterFS apparently doesn't give up 
properly if the unresponsive machine is still up.  After rebooting the 
hung node (but not yet restarting the server processes), the clients gave 
up on the node and became responsive again.

Thanks,

Brent