Re: [Gluster-devel] error while copying files to volume

[Brian Hirt]

I stopped it after I saw the log messages.  When gluster started  
reporting things as errors (instead of warnings) I figured something  
was wrong.   I'll fire it back up and see how it goes.

--brian

On Dec 23, 2008, at 1:03 PM, Anand Avati wrote:

> Brian,
>  These logs related to rsync need not be worried. Is rsync actually
> completing? these log messages will be fixed soon.
>
> Thanks,
> avati
>
> > Hi,
> >
> > Thanks for your help thus far.  I'm continuing my testing and have  
> > run
> > across some errors in my logfiles.   I'm using rsync from a local  
> > directory
> > to the gluster volume to load up some test data.  This is the only
> > processing modifying the gluster filesystem.  Occasionally, I'll see
> > self-heals triggered while the the rsync is running.  With the  
> > exception of
> > this one, they all finish without any errors reported.
> >
> > I thought it was strange that selfheals were being triggered since  
> > all of
> > the servers are up and rsync is the only process modifying the  
> > gluster
> > filesystem, but since it seemed harmless I ignored them.   This  
> > time it
> > seemed to be more of an error.  While rsync is copying files, it  
> > uses a
> > different filename than the file that's actually being copied, and  
> > once the
> > copy is complete, the file is renamed to it's original name.    
> > There seems
> > to have been a race condition here where a self-heal is triggered  
> > and during
> > that time, the file is renamed.   The logs on brick1 and brick3 seem
> > consistent with this theory.   The failed locks seem to make sense  
> > also with
> > the error "No such file or directory"  (the old name is no longer  
> > in use).
> > I see 2 potential problems here.   The first is that self-heals are
> > occasionally being triggered during a copy when everything is up and
> > functioning properly.   The 2nd issue seems to be that during a  
> > self-heal,
> > there is a race condition where a file can be renamed and causes  
> > these
> > errors below.
> >
> > Running 1.4RC3, logs and volfiles are below.  Nothing of mention  
> > was in
> > glusterfs.log except on brick1.  Please let me know what i can do  
> > to help.
> >
> > server x.x.x.102 (brick1)
> > ==> glusterfsd.log <==
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(19824657): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(19824657): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(19824657): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(19824657): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> >
> > ==> glusterfs.log <==
> > 2008-12-18 11:16:20 W [afr-self-heal-common.c:1005:afr_self_heal]  
> > afr-ns:
> > performing self heal on /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc
> > (metadata=1 data=1 entry=1)
> > 2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
> > 686:afr_sh_metadata_lk_cbk]
> > afr-ns: locking of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc on
> > child 3 failed: No such file or directory
> > 2008-12-18 11:16:20 E [afr-self-heal-metadata.c: 
> > 686:afr_sh_metadata_lk_cbk]
> > afr-ns: locking of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc on
> > child 1 failed: No such file or directory
> > 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
> > afr-ns: locking of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc on
> > child 3 failed: No such file or directory
> > 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk]
> > afr-ns: locking of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc on
> > child 1 failed: No such file or directory
> > 2008-12-18 11:16:20 E [afr-self-heal-data.c: 
> > 182:afr_sh_data_unlck_cbk]
> > afr-ns: locking inode of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc
> > on child 1 failed: No such file or directory
> > 2008-12-18 11:16:20 E [afr-self-heal-data.c: 
> > 182:afr_sh_data_unlck_cbk]
> > afr-ns: locking inode of /mobygames/htdocs/images/i/ 
> > 22/23/.139623.png.S254kc
> > on child 3 failed: No such file or directory
> >
> >
> >
> > server x.x.x.106 (brick2)
> > ==> glusterfsd.log <==
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> >
> >
> > server x.x.x107 (brick3)
> > ==> glusterfsd.log <==
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(5570577): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(5570577): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(5570577): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick- 
> > ns: paths
> > differ for inode(5570577): client path =
> > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path =
> > /mobygames/htdocs/images/i/22/23/139623.png
> >
> > server x.x.x.103 (brick4)
> > ==> glusterfsd.log <==
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume]  
> > server:
> > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on  
> > brick-ns
> > returning error: -1 (2)
> >
> >
> > VOLFILES
> >
> > ****Server****
> >
> > volume brick-store
> > type storage/posix
> > option directory /moby/gluster/
> > end-volume
> >
> > volume brick
> > type features/locks
> > subvolumes brick-store
> > end-volume
> >
> > volume brick-store-ns
> > type storage/posix
> > option directory /moby/gluster-ns/
> > end-volume
> >
> > volume brick-ns
> > type features/locks
> > subvolumes brick-store-ns
> > end-volume
> >
> > volume server
> > type protocol/server
> > option transport-type tcp/server
> > subvolumes brick brick-ns
> > option auth.addr.brick.allow x.x.x.* # Allow access to "brick" volume
> > option auth.addr.brick-ns.allow x.x.x.* # Allow access to "brick- 
> > ns" volume
> > end-volume
> >
> >
> > ****Client****
> >
> > volume brick1
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.102
> > option remote-subvolume brick
> > end-volume
> >
> > volume brick2
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.106
> > option remote-subvolume brick
> > end-volume
> >
> > volume brick3
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.107
> > option remote-subvolume brick
> > end-volume
> >
> > volume brick4
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.103
> > option remote-subvolume brick
> > end-volume
> >
> > volume brick-ns1
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.102
> > option remote-subvolume brick-ns
> > end-volume
> >
> > volume brick-ns2
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.106
> > option remote-subvolume brick-ns
> > end-volume
> >
> > volume brick-ns3
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.107
> > option remote-subvolume brick-ns
> > end-volume
> >
> > volume brick-ns4
> > type protocol/client
> > option transport-type tcp/client
> > option remote-host x.x.x.103
> > option remote-subvolume brick-ns
> > end-volume
> >
> > volume afr1
> > type cluster/afr
> > subvolumes brick1 brick3
> > end-volume
> >
> > volume afr2
> > type cluster/afr
> > subvolumes brick2 brick4
> > end-volume
> >
> > volume afr-ns
> > type cluster/afr
> > subvolumes brick-ns1 brick-ns2 brick-ns3 brick-ns4
> > end-volume
> >
> > volume unify
> > type cluster/unify
> > option namespace afr-ns
> > option scheduler rr
> > subvolumes afr1 afr2
> > end-volume
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel