Re: [Gluster-devel] Feedback - Problem with the locks feature

[Samuel Hassine]

Hi all,

For the PHP script with little write/read accesses I will try to find it (I dont remember exactly the syntax), but for PHP Sessions, the bug could be easily reproduced.

I just test it on a new very simple GlusterFS partition with no trafic (juste me), and I reproduced it immediatly.

Explainations:
- 2 servers Debian Lenny stable
- GlusterFS 3.0.0 in distributed mode (one server and multiple clients)
- Lighttpd / PHP5 Fast-CGI

I juste mount the GlusterFS partition on the /var/www directory.

First of all, the PHP script you can execute:

<?php
session_save_path('.');
//if you want to verify if it worked
//echo session_save_path();
session_start();
?>

Secondly, there are 2 configurations if GlusterFS and, of course, one works and one does not.
The client configuration is the same in the both cases:

glusterfs.vol
volume test-1
    type protocol/client
    option transport-type tcp
    option remote-host test
    option transport.socket.nodelay on
    option transport.remote-port 6996
    option remote-subvolume brick1
end-volume

volume writebehind
    type performance/write-behind
    option cache-size 4MB
    subvolumes test-1
end-volume

volume readahead
    type performance/read-ahead
    option page-count 4
    subvolumes writebehind
end-volume

volume iocache
    type performance/io-cache
    option cache-size 1GB
    option cache-timeout 1
    subvolumes readahead
end-volume

volume quickread
    type performance/quick-read
    option cache-timeout 1
    option max-file-size 64kB
    subvolumes iocache
end-volume

volume statprefetch
    type performance/stat-prefetch
    subvolumes quickread
end-volume

Now the server configuration:

glusterfsd.vol (this doesnt work)
volume posix1
  type storage/posix
  option directory /data
end-volume

volume locks1
    type features/locks
    subvolumes posix1
end-volume

volume brick1
    type performance/io-threads
    option thread-count 8
    subvolumes locks1 
end-volume

volume server-tcp
    type protocol/server
    option transport-type tcp
    option auth.addr.brick1.allow *
    option transport.socket.listen-port 6996
    option transport.socket.nodelay on
    subvolumes brick1
end-volume

glusterfsd.vol (this works)
volume posix1
  type storage/posix
  option directory /data
end-volume

#volume locks1
#    type features/locks
#   subvolumes posix1
#end-volume

volume brick1
    type performance/io-threads
    option thread-count 8
    subvolumes posix1 
end-volume

volume server-tcp
    type protocol/server
    option transport-type tcp
    option auth.addr.brick1.allow *
    option transport.socket.listen-port 6996
    option transport.socket.nodelay on
    subvolumes brick1
end-volume

So, with the locks translator, you can execute the script one time (it will be ok) but the second time the session file is on the file system but locked and nobody can access to it. PHP freezes and processes coult not be killed.

When it's happened, I have nothing in client-side logs but I have 2 kinds of message in the server-side logs:
When I execute the script:
[2010-02-04 21:11:22] W [posix.c:246:posix_lstat_with_gen] posix1: Access to /data//.. (on dev 2049) is crossing device (64768)
[2010-02-04 21:11:24] W [posix.c:246:posix_lstat_with_gen] posix1: Access to /data//.. (on dev 2049) is crossing device (64768)

When I try to umount -f (disconnect the gluster):
[2010-02-04 21:13:45] E [server-protocol.c:339:protocol_server_reply] protocol/server: frame 20: failed to submit. op= 26, type= 4

As I said I will try to find the other PHP script.

I hope this will help you.

Thanks for your hard work, GlusterFS is a great project.

Regards.

Le jeudi 04 février 2010 à 11:25 -0600, Tejas N. Bhise a écrit :
> Hi Samuel,
>
> This problem is important for us to fix. We have heard of this from 2-3 users. It seems to be a combination of apache, php session files shared by multiple apache servers and locking ( or lock translator ). 
>
> We have tried very hard to reproduce this with test programs that read/write very small files across multiple clients ( to mimic behaviour of shared php session files ). We even tried this using sample programs ( even jmeter ) using php sessions - but we were not able to reproduce it.
>
> Are you able to reproduce it easily ? If so, we would request you to help us reproduce this inhouse or else help us with some tracing from your system - ( disclaimer - the tracing could slow down the system a bit :-) ) ..
>
> Maybe we can start with using your test script - please let us know what gluster config, machine config/type etc you used so we can try something as close as possible.
>
> Thank you for your help in making the product better.
>
> Regards,
> Tejas.
>
>
>
>
> ----- Original Message -----
> From: "Samuel Hassine" <address@hidden>
> To: "Gluster List" <address@hidden>
> Cc: "Yann Autissier" <address@hidden>
> Sent: Thursday, February 4, 2010 8:14:34 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
> Subject: [Gluster-devel] Feedback - Problem with the locks feature
>
> Hi all,
>
> Just a little feedback about a special using of GlusterFS.
>
> We are hosting about 15 000 websites on an infrastructure containing a
> big files server. We just change our files sharing system from NFS to
> GlusterFS. So we are using a simple distributed GlusterFS for the
> websites files. We are alo using a second GlusterFSd instance (on an
> alternate port) in order to export a special partition for PHP sessions.
>
> On the first partition, it seems to work very well for reading and
> writing. But on the second one, with a "classic" server configuration
> (posix vol, locks vol, performance vol and server vol), the file system
> freezes immediatly. After some tests, we found that it works fine
> without the locks volume.
>
> But after some other tests on the first GlusterFS, we discover that the
> same problem could occured with a script that writes many little files
> on the FS. It is the same problem with the last version of ezpublish,
> the cache generation failed).
>
> Do you think it could be fixed in the next versions or just bypassing
> the locks translator is a good solution?
>
> Thanks for your answers.
> Sam
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/gluster-devel