Hi!
I have errors in my application caused by GlusterFS:
copy(/storage/2060000/2064000/2064000/2064011/dat): failed to open stream: Input/output error
readfile(/storage/2060000/2064000/2064000/2064011/dat): failed to open stream: Input/output error
$ cat /storage/2120000/2125000/2125800/2125844/25/dat
cat: /storage/2120000/2125000/2125800/2125844/25/dat: Input/output error
On client Instance:
$ tail /var/log/gluster.log
[2012-10-25 07:17:14.904162] W [afr-self-heal-data.c:831:afr_lookup_select_read_child_by_txn_type] 0-storage-replicate-0: /2120000/2125000/2125800/2125844/25/game.dat: Possible split-brain
[2012-10-25 07:17:14.904225] W [afr-common.c:1226:afr_detect_self_heal_by_lookup_status] 0-storage-replicate-0: split brain detected during lookup of /2120000/2125000/2125800/2125844/25/game.dat.
[2012-10-25 07:17:14.911705] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-storage-replicate-0: background data gfid self-heal failed on /2120000/2125000/2125800/2125844/25/game.dat
[2012-10-25 07:17:14.911885] W [afr-open.c:213:afr_open] 0-storage-replicate-0: failed to open as split brain seen, returning EIO
[2012-10-25 07:17:14.912047] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 457439: OPEN() /2120000/2125000/2125800/2125844/25/game.dat => -1 (Input/output error)
[2012-10-25 07:17:22.386426] W [afr-self-heal-data.c:831:afr_lookup_select_read_child_by_txn_type] 0-storage-replicate-0: /2120000/2125000/2125800/2125844/25/game.dat: Possible split-brain
[2012-10-25 07:17:22.386469] W [afr-common.c:1226:afr_detect_self_heal_by_lookup_status] 0-storage-replicate-0: split brain detected during lookup of /2120000/2125000/2125800/2125844/25/game.dat.
[2012-10-25 07:17:22.393897] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-storage-replicate-0: background data gfid self-heal failed on /2120000/2125000/2125800/2125844/25/game.dat
[2012-10-25 07:17:22.397818] W [afr-open.c:213:afr_open] 0-storage-replicate-0: failed to open as split brain seen, returning EIO
[2012-10-25 07:17:22.397853] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 459320: OPEN() /2120000/2125000/2125800/2125844/25/game.dat => -1 (Input/output error)
$ glusterd -V
glusterfs 3.3.0 built on Jun 4 2012 11:39:45
$ gluster volume info
Volume Name: storage
Type: Distributed-Replicate
Volume ID: a278889e-16fe-4913-9af6-eca4c099c484
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: gluster-1a:/storage/1a
Brick2: gluster-1b:/storage/1b
Brick3: gluster-2a:/storage/2a
Brick4: gluster-2b:/storage/2b
Brick5: gluster-1a:/storage/3a
Brick6: gluster-1b:/storage/3b
Brick7: gluster-2a:/storage/4a
Brick8: gluster-2b:/storage/4b
Options Reconfigured:
performance.stat-prefetch: off
auth.allow: 10.*
$ gluster volume status
Status of volume: storage
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-1a:/storage/1a 24009 Y 17302
Brick gluster-1b:/storage/1b 24009 Y 12188
Brick gluster-2a:/storage/2a 24009 Y 10863
Brick gluster-2b:/storage/2b 24009 Y 13486
Brick gluster-1a:/storage/3a 24011 Y 17308
Brick gluster-1b:/storage/3b 24011 Y 12194
Brick gluster-2a:/storage/4a 24011 Y 10869
Brick gluster-2b:/storage/4b 24011 Y 13492
NFS Server on localhost 38467 Y 17314
Self-heal Daemon on localhost N/A Y 17320
NFS Server on gluster-2a 38467 Y 10879
Self-heal Daemon on gluster-2a N/A Y 10885
NFS Server on gluster-1b 38467 Y 12200
Self-heal Daemon on gluster-1b N/A Y 12206
NFS Server on gluster-2b 38467 Y 13503
Self-heal Daemon on gluster-2b N/A Y 13509
How can I fix the situation?