Re: [Gluster-devel] Problem with GlusterFS quotas (Debian Wheezy/x86_64)

[Hristo Hristov]

Hello Raghavendra,

Thank you for your replay /and appropriate patch-set(s) :)/. Let me 
summarize the results I achieved during the testing. Testing environment 
was the same: two nodes, one replicated volume. Here is brief 
description of my test-bet volume.

=== snip ===
muzzy:~# gluster volume info m2d

Volume Name: m2d
Type: Replicate
Volume ID: 4d29926d-38d7-4b41-b500-83875ff37c44
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: g0:/g0
Brick2: g1:/g1
Options Reconfigured:
auth.allow: 194.12.*,130.204.198.50,82.119.73.118
features.quota: on
features.limit-usage: 
/dav/64mb:64MB,/cifs/64mb:64MB,/cifs/384mb:384MB,/cifs/352mb:352MB

muzzy:~# gluster volume quota m2d list
    path          limit_set         size
----------------------------------------------------------------------------------
/dav/64mb                  64MB               30.2MB
/cifs/64mb                 64MB               42.2MB
/cifs/384mb               384MB              365.5MB
/cifs/352mb               352MB              335.3MB

g0:
muzzy:~# du -sh /g0/dav/64mb
31M    /g0/dav/64mb
muzzy:~# du -sh /g0/cifs/64mb
43M    /g0/cifs/64mb
muzzy:~# du -sh /g0/cifs/384mb
366M    /g0/cifs/384mb
muzzy:~# du -sh /g0/cifs/352mb
336M    /g0/cifs/352mb

g1:
address@hidden:/# du -sh /g1/dav/64mb
31M    /g1/dav/64mb
address@hidden:/# du -sh /g1/cifs/64mb
43M    /g1/cifs/64mb
address@hidden:/# du -sh /g1/cifs/384mb
366M    /g1/cifs/384mb
address@hidden:/# du -sh /g1/cifs/352mb
336M    /g1/cifs/352mb
=== snip ===

Basically I did again my quota testing in both (already described) ways, 
i.e. with quota limit set on existing and on non-existing directory 
entry. Everything went ok this time. Quota limits correctly take effect 
and client sides are notified correctly.

Let me describe some more findings during the testing:

(o) patches applied to master glusterfs branch:

- a compile time issue: after execution of autogen.sh script, where was 
successfully generated configure scrip. When I tried to use the script, 
the following error occured:

=== snip ===
checking if libxml2 is present... ./configure: line 12771: syntax error 
near unexpected token `LIBXML2,'
./configure: line 12771: `PKG_CHECK_MODULES(LIBXML2, libxml-2.0 >= 2.6.19,'
=== snip ===

After fixing broken check, I had successful compilation and 
installation.  During, my testing with master branch, I noticed the 
following:

- there was a huge slow down of file transfer between g0 serving node 
and my testing webdav client;
- there was a strange quota size variation during the upload;
- although these problems quota limit restriction successfully took place;

(o) patches applied to qa37 tar:

- no compile time issues;
- i did not notice any performance issues;
- quota lime worked like expected;

So, for now I'm stuck on 3.3.0qa37 + described patches :)

Please find attached glusterd logs from two nodes (g0 and g1) and native 
client log from g0 node. All logs are taken with DEBUG option on.

I've planed more tests - with distributed and replicated volume. I'll 
test again with patched 3.3.0qa37  version and I hope to achieve same 
results :) during the testing.

Regards,
Hristo Hristov

On 4/25/12 5:14 AM, Raghavendra Gowdappa wrote:
> Hi,
>
> Can you apply following patch to latest master?
> http://review.gluster.com/3022
>
> and let us know, whether it solves your issue?
>
> If you want to stick to 3.3.0qa37, you need following patch along with the 
> above:
> http://review.gluster.com/#change,3021
>
> regards,
> Raghavendra.
>
> ----- Original Message -----
> > From: "Hristo Hristov"<address@hidden>
> > To: "Vijay Bellur"<address@hidden>
> > Cc: address@hidden, "Raghavendra Gowdappa"<address@hidden>
> > Sent: Monday, April 23, 2012 7:25:35 PM
> > Subject: Re: [Gluster-devel] Problem with GlusterFS quotas (Debian 
> > Wheezy/x86_64)
> >
> > Hello Vijay,
> >
> > I made the testing against the version 3.3.0 qa 37 of gluster fs with
> > same cluster config. I'm pasting configuration info:
> >
> > === snip ===
> > muzzy:/home/lisp/work/fs/logs# gluster --version
> > glusterfs 3.3.0qa37 built on Apr 23 2012 13:55:08
> > Repository revision: git://git.gluster.com/glusterfs.git
> > Copyright (c) 2006-2011 Gluster Inc.<http://www.gluster.com>
> > GlusterFS comes with ABSOLUTELY NO WARRANTY.
> > You may redistribute copies of GlusterFS under the terms of the GNU
> > General Public License.
> >
> > muzzy:/home/lisp/work/fs/logs# gluster volume info m2d
> >
> > Volume Name: m2d
> > Type: Replicate
> > Volume ID: 752e6b5b-504b-46bc-be5e-498fc5f364d1
> > Status: Started
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: g0:/g0
> > Brick2: g1:/g1
> > Options Reconfigured:
> > features.limit-usage:
> > /dav/64mb:64MB,/cifs/64mb:64MB,/dav/92mb:92MB,/cifs/92mb:92MB,/cifs/128mb:128MB,/dav/128mb:128MB
> > features.quota: on
> > auth.allow: 194.12.*
> > muzzy:/home/lisp/work/fs/logs#
> >
> > muzzy:/home/lisp/work/fs/logs# gluster volume quota m2d list
> >        path          limit_set         size
> > ----------------------------------------------------------------------------------
> > /dav/64mb                  64MB              365.5MB
> > /cifs/64mb                 64MB              365.5MB
> > /dav/92mb                  92MB              365.5MB
> > /cifs/92mb                 92MB               0Bytes
> > /cifs/128mb               128MB              126.9MB
> > /dav/128mb                128MB               83.6MB
> >
> >
> > mount -t glusterfs g0:/m2d /mnt/m2d -o
> > acl,log-level=DEBUG,log-file=/usr/local/var/log/glusterfs/g0-mount-client.log
> > === snip ===
> >
> > Unfortunately, gluster's quota management behaves exactly the same
> > way,
> > like in previous version. When I set a quota limit (no matter on
> > existing or not existing directory), quota limitation doesn't work,
> > until net next gluster volume remount (i.e. unmount and then mount)
> >
> > Please find attached log files from g0 node:
> >
> > (o) g0-daemon.log
> > log from glusterfs daemon, run in debug mode;
> >
> > (o) 0-mount-client.log.00.before-remounting
> > client log before remounting;
> >
> > (o) g0-mount-client.log.01.after-remounting
> > client log after unmount, mount (i.e. remounting) sequence; then
> > quota
> > restrictions work as expected (Entries '/cifs/128mb' and '/dav/128mb'
> > in
> > upper 'gluster volume quota m2d list' quota).
> >
> > All the log files are inside 3-3-0-qa-38-logs.tar.gz archive.
> >
> > One more thing, could you please point me out something like a
> > 'big-picture-of-glusterfs-architecture' page/doc @ gluster's web
> > site.
> > I'm constantly finding a
> > http://www.gluster.org/community/documentation/index.php/GlusterFS_Hackers_Guide
> > link, which is quite outdated :(
> >
> > Regards,
> > Hristo Hristov
> >
> > On 4/23/12 1:26 PM, Vijay Bellur wrote:
> > > On 04/23/2012 01:17 PM, Hristo Hristov wrote:
> > > > Thank you for your replay. I'm sending you requested files, please
> > > > correct me if there is a missing one :) /debian logs its services
> > > > in
> > > > debianish way, so I may be missing something:)/
> > > >
> > > > (o) glusterd log from brick 0:
> > > >
> > > > location: /var/log/glusterfs/bricks
> > > > name: g0.log
> > > >
> > > > (o) native client mount log
> > > >
> > > > name: gluster.log
> > > >
> > > > I'll definitely try v3.3.0 release of glusterfs with same test
> > > > configuration. Which version I should try (i.e. which one is more
> > > > recent):
> > > >
> > > > (o)
> > > > http://bits.gluster.com/pub/gluster/glusterfs/src/glusterfs-3.3.0qa37.tar.gz
> > > qa37 is more recent. That would be a good release to try this out
> > > on.
> > >
> > > Thanks,
> > > Vijay

qa37.patched.tar.gz
Description: GNU Zip compressed data