gluster-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] [Gluster-users] Self Heal and dangling symlinks


From: Alexandre Fournier
Subject: Re: [Gluster-devel] [Gluster-users] Self Heal and dangling symlinks
Date: Thu, 21 Nov 2013 17:43:19 -0500

I would like also to inform you about the information about the split-brain we 
have : 

2013-10-09 22:02:59 <gfid:83b39e48-f8eb-4149-b851-d3b97e18c4b6>
2013-10-09 21:52:59 <gfid:85378e3f-0dd1-4f8e-a7d5-70424b643fb9>
2013-10-09 21:52:59 <gfid:0a958cad-1615-4e1b-8e1a-9dc0356859d6>
2013-10-09 21:52:59 <gfid:54b02fde-69d2-4da2-8372-2a7af89a0ae1>
2013-10-09 21:52:59 <gfid:4702c3ab-a2bb-43e3-ae2c-ecb5b440f368>
2013-10-09 21:52:59 <gfid:8fe46824-a9f1-4095-b204-e9e137ae8643>
...
Count : 1023

We tried to clean all the dangling links but they are still coming back the 
split-brain is not resolved.

It maybe the root cause of the problem, how do we resolve those split brain?

-----Original Message-----
From: Alexandre Fournier 
Sent: 21 novembre 2013 15:20
To: 'Lalatendu Mohanty'; Pranith Kumar Karampuri
Cc: address@hidden; address@hidden
Subject: RE: [Gluster-devel] [Gluster-users] Self Heal and dangling symlinks

Ok here is the information : 

Stat : 
  File: `/aa/aa/aa/aa/aa/aa/aa
  Size: 14364           Blocks: 32         IO Block: 4096   regular file
Device: 822h/2082d      Inode: 3155137861  Links: 2
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2013-11-21 13:14:58.527765935 +0000
Modify: 2013-11-13 13:19:13.736226050 +0000
Change: 2013-11-13 13:19:13.736226050 +0000

  File: `/aa/aa/aa/aa/aa/aa/aa
  Size: 14364           Blocks: 32         IO Block: 4096   regular file
Device: 822h/2082d      Inode: 3076494286  Links: 2
Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2013-11-21 13:14:58.527674754 +0000
Modify: 2013-11-13 13:19:13.736442464 +0000
Change: 2013-11-13 13:19:13.736442464 +0000
 Birth: -

Attributes :

# file: aa/aa/aa/aa/aa/aa/aa
trusted.afr.gv0-client-0=0x000000000000000000000000
trusted.afr.gv0-client-1=0x000000000000000000000000
trusted.gfid=0xb5b8c3ec9dd24609b56476651113d3fa


# file: aa/aa/aa/aa/aa/aa/aa
trusted.afr.gv0-client-0=0x000000000000000000000000
trusted.afr.gv0-client-1=0x000000000000000000000000
trusted.gfid=0xb5b8c3ec9dd24609b56476651113d3fa


-----Original Message-----
From: Lalatendu Mohanty [mailto:address@hidden
Sent: 21 novembre 2013 14:05
To: Alexandre Fournier; Pranith Kumar Karampuri
Cc: address@hidden; address@hidden
Subject: Re: [Gluster-devel] [Gluster-users] Self Heal and dangling symlinks

On 11/21/2013 07:54 PM, Alexandre Fournier wrote:
> They are both regular file on the node and the replicas and they have 
> the same GFID.  I ran also the gluster volume heal gv0 split-brain 
> command and the file is not in the list.  We have an entire directory 
> though (1023 entry on a node)
>
> However, the file was already on the brick before uploading it and I noticed 
> that that the write did not work since the last modification date does not 
> match the upload time.
>
> Through a web service, we offer to upload files on the gluster mount.   This 
> web service  upload the file on a temporary folder and then MOVE the file on 
> the gluster mount.
>
> Could the move operation give strange behavior like this?
Alexandre,

No, it should not. Please let us know the answers of the questions Pranith and 
I asked, so we can understand the root cause of your problem.

>
> Alexandre Fournier
> Tools Programmer
> Ubisoft Production Services
>
>
> -----Original Message-----
> From: Pranith Kumar Karampuri [mailto:address@hidden
> Sent: 21 novembre 2013 00:47
> To: Lalatendu Mohanty
> Cc: Alexandre Fournier; address@hidden; 
> address@hidden
> Subject: Re: [Gluster-devel] [Gluster-users] Self Heal and dangling 
> symlinks
>
> Alexandre,
>     Seems like there is an entry split-brain (same file/dir name but on one 
> brick it is a file and on the other it is a directory) according to the 
> following log:
>> [2013-11-18 18:18:43.052446] W
>> [afr-common.c:1411:afr_conflicting_iattrs]
>> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0,
>> 1)
> Could you get us the output of "stat <brick-dir-path>/aa/aa/aa/aa/aa" and 
> "getfattr -d -m. -e hex <brick-dir-path>/aa/aa/aa/aa/aa" on both the bricks.
>
> Pranith
> ----- Original Message -----
>> From: "Lalatendu Mohanty" <address@hidden>
>> To: "Alexandre Fournier" <address@hidden>, 
>> address@hidden, address@hidden
>> Sent: Thursday, November 21, 2013 1:28:01 AM
>> Subject: Re: [Gluster-devel] [Gluster-users] Self Heal and dangling 
>> symlinks
>>
>> On 11/19/2013 10:49 PM, Alexandre Fournier wrote:
>>
>>
>>
>>
>>
>> Hello,
>>
>>
>>
>> We are experiencing strange behavior when writing file on the Gluster 
>> mount point. On some occasion, when writing to the Gluster Mount we 
>> have an Open Stream error. We’ve looked the gluster logs and found 
>> the following faulty entries :
>>
>>
>>
>> [From /var/log/glusterfs/mnt-gv0.log]
>>
>>
>>
>> [2013-11-18 18:18:43.052446] W
>> [afr-common.c:1411:afr_conflicting_iattrs]
>> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0,
>> 1)
>>
>> [2013-11-18 18:18:43.052468] E
>> [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0:
>> Conflicting entries for /aa/aa/aa/aa
>>
>> [2013-11-18 18:18:43.052757] E
>> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
>> 0-gv0-replicate-0: background meta-data data entry missing-entry gfid 
>> self-heal
>>
>> failed on /aa/aa/aa/aa/aa
>>
>> [2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk]
>> 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 
>> (Input/output
>> error)
>>
>>
>>
>> We’ve looked at the log file etc-glusterfs-glusterd.vol.log but we 
>> found nothing related to this problem. Then, we’ve looked at the log 
>> From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of 
>> logs of the same type :
>>
>>
>>
>> [2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft]
>> 0-gv0-posix: symlink
>> ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> 
>> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf
>> failed (No such file or directory)
>>
>> [2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft]
>> 0-gv0-posix: symlink
>> ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 ->
>> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2
>> failed (No such file or directory)
>>
>> [2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft]
>> 0-gv0-posix: symlink
>> ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 ->
>> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2
>> failed (No such file or directory)
>>
>>
>>
>> This looks like a bug, unless there is something wrong with the 
>> set-up. I have copied gluster-devel in this thread as I think they might 
>> help.
>>
>> Just curious, is all your gluster nodes have equal time (i.e. ntp synced).
>>
>>
>>
>>
>>
>>
>> And it does not stop logging. It seems that the self heal is not 
>> working properly when there are broking symlinks in the gluster. It 
>> is worth saying also that this log is only produce on a single node 
>> but the write fail on several node though. Also, we try to clean the 
>> symlinks manually but it always come back.
>>
>>
>>
>> Is it possible to recover from broken symlinks?
>>
>>
>>
>> Configuration :
>>
>> Gluster Version : 3.3.2
>>
>> Cluster setup : 4 X 2
>>
>> OS : Ubuntu
>>
>> On Fuse
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Alexandre
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list address@hidden 
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>>


reply via email to

[Prev in Thread] Current Thread [Next in Thread]