[Gluster-devel] split brain

[Emmanuel Dreyfus]

After I added bricks, I have one server that is always busy trying to heal the
volume, and it gets a lot of split brain errors. glustershd.log says:

[2012-08-15 16:54:30.562148] I [afr-common.c:1340:afr_launch_self_heal]
   0-gfs33-replicate-0: background  meta-data self-heal triggered. path: 
  <gfid:29d170bb-6391-40ed-b4c6-27d8caa72a64>, reason: lookup detected 
  pending operations
[2012-08-15 16:54:30.693520] E  [afr-self-heal-metadata.c:481: 
  afr_sh_metadata_fix] 0-gfs33-replicate-0: Unable to self-heal
  permissions/ownership of '<gfid:29d170bb-6391-40ed-b4c6-27d8caa72a64>'    
  (possible split-brain). Please fix the file on all backend volumes
[2012-08-15 16:54:30.694065] I [afr-self-heal-metadata.c:56:
  afr_sh_metadata_done] 0-gfs33-replicate-0: split-brain detected,
  aborting selfheal of <gfid:29d170bb-6391-40ed-b4c6-27d8caa72a64>

Is there a simplier way of retreiving of offending file that looking up
.glusterfs/29/d1/29d170bb-6391-40ed-b4c6-27d8caa72a64, get the inode, and
search it in the entiere brick?

Here is what I find:

# ls -l /export/*/.glusterfs/29/d1/29d170bb-6391-40ed-b4c6-27d8caa72a64
lrwxrwxrwx  1 root  wheel  55 Aug 10 18:47
/export/wd3a/.glusterfs/29/d1/29d170bb-6391-40ed-b4c6-27d8caa72a64 ->
../../2e/23/2e231351-fc16-46ef-b38b-1d5f10ffd5d5/libbfd

# ls -dil
/export/*/.glusterfs/2e/23/2e231351-fc16-46ef-b38b-1d5f10ffd5d5/libbfd
50951712 drwxr-xr-x  4 manu  manu  2560 Aug 13 17:40
/export/wd3a/.glusterfs/2e/23/2e231351-fc16-46ef-b38b-1d5f10ffd5d5/libbfd

# find /export/wd3a/ -inum 50951712 -exec ls -ldi {} \; 
50951712 drwxr-xr-x  4 manu  manu  2560 Aug 13 17:40
/export/wd3a/manu/netbsd/usr/src/gnu/lib/libbfd

Attributes:
trusted.glusterfs.dht         00 00 00 01 00 00 00 00 7f ff ff ff ff ff ff ff
trusted.afr.gfs33-client-1  00 00 00 00 00 00 00 02 00 00 00 00
trusted.afr.gfs33-client-0  00 00 00 00 00 00 00 00 00 00 00 00  
trusted.gfid                   29 d1 70 bb 63 91 40 ed b4 c6 27 d8 ca a7 2a 64

On the other bricks:
trusted.glusterfs.dht         00 00 00 01 00 00 00 00 00 00 00 00 7f ff ff fe
trusted.afr.gfs33-client-2   00 00 00 00 00 00 00 00 00 00 00 00
trusted.afr.gfs33-client-3  00 00 00 00 00 00 00 00 00 00 00 00  
trusted.gfid                   29 d1 70 bb 63 91 40 ed b4 c6 27 d8 ca a7 2a 64

trusted.glusterfs.dht         00 00 00 01 00 00 00 00 7f ff ff ff ff ff ff ff
trusted.afr.gfs33-client-1  00 00 00 00 00 00 00 00 00 00 00 00
trusted.afr.gfs33-client-3  00 00 00 00 00 00 00 00 00 00 00 00  
trusted.gfid                   29 d1 70 bb 63 91 40 ed b4 c6 27 d8 ca a7 2a 64

trusted.glusterfs.dht         00 00 00 01 00 00 00 00 00 00 00 00 7f ff ff fe
trusted.afr.gfs33-client-2   00 00 00 00 00 00 00 01 00 00 00 00 
trusted.afr.gfs33-client-3  00 00 00 00 00 00 00 00 00 00 00 00  
trusted.gfid                   29 d1 70 bb 63 91 40 ed b4 c6 27 d8 ca a7 2a 64

I tried to understand the code here, It is reading trusted.afr.gfs33-client-*
and it builds a matrix, which looks like this:
pending_matrix: [ 0 1 ]
pending_matrix: [ 2 0 ]

Then afr_sh_wise_nodes_conflict() decides that nsources = -1. 

Is there some documentation explaining how it works? Someone call tell me why
it decides it is split brain?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
address@hidden