Re: Re:[Gluster-devel] NFS re-export works for me

gluster-devel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Re:[Gluster-devel] NFS re-export works for me

From:	Brent A Nelson
Subject:	Re: Re:[Gluster-devel] NFS re-export works for me
Date:	Mon, 9 Mar 2009 17:37:36 -0400 (EDT)
User-agent:	Alpine 1.00 (DEB 882 2007-12-20)

Note that you can also use the UNFS3 user-mode NFS server rather than thenfs-kernel-server. With it, you don't need a custom FUSE, and you don'tneed to disable direct-IO. Probably the big downside, though, is that itdoesn't support NFS file locking.


Thanks,

Brent

On Mon, 9 Mar 2009, Raghavendra G wrote:

Hi,

starting glusterfs with direct-io disabled (glusterfs --disable-direct-io)
ensures working of NFS re-export of glusterfs.

regards,

On Fri, Apr 27, 2007 at 12:47 AM, address@hidden <address@hidden> wrote:

Hi Avati,

I first thought that I found the error when suddenly it was
working on my machine but I never managed to reproduce it.
Then I realized that in cluster mode, one of my machine was
running using the default fuse but it did it again on my
standalone machine.
One think for sure is that with default fuse (API 7.6)
NFS mount fail always. You have to use a latter fuse.

After many try I started to trace NFS and Fuse in kernel mode:

cd /proc/sys/sunrpc
echo 32767 > nfs_debug
echo 32767 > nfsd_debug

and did my test again:

exportfs:
localhost:/gfs/fs1 on /gfs/d610.nfsV3 type nfs
(rw,nfsvers=3,noac,hard,intr,tcp,lock,posix,addr=127.0.0.1)

cd /gfs/nfsV3
ls > 11


The trace between a successfull writing other NFS and error
change here:
Apr 26 16:21:06 fturi kernel: NFS call  write 6 @ 0
Apr 26 16:21:06 fturi kernel: nfsd_dispatch: vers 3 proc 7
Apr 26 16:21:06 fturi kernel: nfsd: WRITE(3)    24: 02010001
00000000 00000003 00000000 00000001 00000000 6 bytes at 0 stable
Apr 26 16:21:06 fturi kernel: nfsd: fh_verify(24: 02010001
00000000 00000003 00000000 00000001 00000000)

Apr 26 16:21:06 fturi kernel: # NO PERM 531 nfsd: write
complete err=-1
Apr 26 16:21:06 fturi kernel: NFS reply write: -1
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

The NO PERM message comes from this function in fuse
2.6.3/kernel/file.c

static int fuse_get_user_pages(struct fuse_req *req, const
char __user *buf,
                              unsigned nbytes, int write)
{
       unsigned long user_addr = (unsigned long) buf;
       unsigned offset = user_addr & ~PAGE_MASK;
       int npages;

       /* This doesn't work with nfsd */
       if (!current->mm)
               {printk(KERN_CRIT "# NO PERM
%d",__LINE__);return -EPERM;};

       nbytes = min(nbytes, (unsigned)
FUSE_MAX_PAGES_PER_REQ << PAGE_SHIFT);
       npages = (nbytes + offset + PAGE_SIZE - 1) >>
PAGE_SHIFT;
       npages = max(npages, 1);
       npages = min(npages, FUSE_MAX_PAGES_PER_REQ);
       down_read(&current->mm->mmap_sem);
       npages = get_user_pages(current, current->mm,
user_addr, npages, write,
                               0, req->pages, NULL);
       up_read(&current->mm->mmap_sem);
       if (npages < 0)
               return npages;

       req->num_pages = npages;
       req->page_offset = offset;
       return 0;
}


I Just added the dprintk to make sure it was there. There is
a comment about: /* doesnt work with nfsd */

So, its look like its fuse refusing (error=-1 = NOPERM) the
writting other NFS.
I'm puzzled by the fact it works under another linux. maybe
the fuse_get_user_pages  is not called in your case of the
structure "current->mm" is filled up in your case.

This is the trace collected by the debug mode of server.vol
and client.vol ( debug on top of the brick):

glusterfsd is empty
glusterfs contains:
[Apr 26 16:21:02] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:02] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:21:04] [DEBUG/trace.c:967/trace_open()]
trace:trace_open (*this=0x8395650, path=/11, flags=32769,
mode=0)
[Apr 26 16:21:04] [DEBUG/trace.c:88/trace_open_cbk()]
trace:trace_open_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *ctx=0x8396f58), *buf=0x8396c60 {st_dev=64768,
st_ino=
64356, st_mode=33206, st_nlink=1, st_uid=99, st_gid=99,
st_rdev=0, st_size=0, st_blksize=4096, st_blocks=8,
st_atime=[Apr 26 15:48:19], st_mtime=[Apr 26 15:54:32], st_cti
me=[Apr 26 15:54:32]})
[Apr 26 16:21:04] [DEBUG/trace.c:1089/trace_release()]
trace:trace_release (*this=0x8395650, *ctx=0x8396f58)
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/11)

[Apr 26 16:21:04] [DEBUG/trace.c:386/trace_release_cbk()]
trace:trace_release_cbk (*this=0x8395650, op_ret=0, op_errno=95)
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8397098 {st_dev=64768, st_ino=64356, st_
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
 15:54:32]})
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
trace:trace_getattr (*this=0x8395650, path=/)

[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
r 26 09:42:58]})
[Apr 26 16:28:31] [DEBUG/trace.c:1155/trace_getxattr()]
trace:trace_getxattr (*this=0x8395650, path=/,
name=system.posix_acl_access, size=0)
[Apr 26 16:28:31] [DEBUG/trace.c:513/trace_getxattr_cbk()]
trace:trace_getxattr_cbk (*this=0x8395650, op_ret=-1,
op_errno=95)


Cdt FT





---------- Initial Header -----------

From      : Anand Avati <address@hidden>
To          : address@hidden
Cc          : address@hidden
Date      : Wed, 18 Apr 2007 08:55:26 -0700
Subject : Re: Re:[Gluster-devel] NFS re-export works for me

I was able to successfully mount a glusterfs export via NFS
too. I am
using slackware 10.2 and did nothing special apart from using
fuse-2.6.0. Maybe your problem could be because of the AIX
client.
please try from a GNU/Linux client and see if it helps.
Attaching the
log file by loading debug/trace will help get a better picture.

regards,
avati

On Wed, Apr 18, 2007 at 11:25:49AM -0400, Brent A Nelson wrote:

rw,fsid=10 are the export options.

2.6.15 is the kernel (it's a Ubuntu Dapper with its

kernel, probably

closer to 2.6.16).  Fuse 2.6.3 was added from an extra

repository.


I used no options at all on the Ubuntu Breezy NFS client

mount.  Looking

at the mtab, no special options are listed there, either,

so the options

are whatever is default for this platform.

Thanks,

Brent

On Wed, 18 Apr 2007, address@hidden wrote:

Hi Brent,

That's a very good news for me. Could you please tell me
which linux kernel version are you using (uname -a) and
/etc/exports line and mount command you used ?
I'm not lucky with NFS and will very appreciate thoses

details.


Have a nice day, Francois
---------- Initial Header -----------

From      : address@hidden
To          : address@hidden
Cc          :
Date      : Wed, 18 Apr 2007 00:51:51 -0400 (EDT)
Subject : [Gluster-devel] NFS re-export works for me

I tried a quick NFS re-export to another Linux box (with the
fsid= option
and fuse 2.6.3 on the NFS server, of course).  I didn't test
it much, but
copies to and from the NFS filesystem worked just fine.  I
think writes
were very slow, but they didn't give me any trouble.

No special mount options (such as NFS version or transport)
were needed.

Thanks,

Brent


_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel


------------------------ ALICE C'EST ENCORE MIEUX AVEC

CANAL+ LE BOUQUET !

---------------
Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE

BOUQUET, en cliquant

ici http://alicebox.fr
Soumis à conditions.

_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel



--
ultimate_answer_t
deep_thought (void)
{
 sleep (years2secs (7500000));
 return 42;
}


------------------------ ALICE C'EST ENCORE MIEUX AVEC CANAL+ LE BOUQUET !
---------------
Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE BOUQUET, en cliquant
ici http://alicebox.fr
Soumis à conditions.




_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel




--
Raghavendra G

[Prev in Thread]

Current Thread

[Next in Thread]

Re: Re:[Gluster-devel] NFS re-export works for me, Raghavendra G, 2009/03/09
- Re: Re:[Gluster-devel] NFS re-export works for me, Brent A Nelson <=

Prev by Date: Re: [Gluster-devel] Memory Leak
Next by Date: Re: [Gluster-devel] 2.0.0rc4 broken transport-type unix
Previous by thread: Re: Re:[Gluster-devel] NFS re-export works for me
Next by thread: [Gluster-devel] ping timer expired! bailing transport
Index(es):
- Date
- Thread