[Gluster-devel] Disconnections and Corruption Under High Load

[Gordan Bobic]

I've noticed a very high incidence of the problem I reported a while 
back, that manifests itself in open files getting corrupted on commit, 
possibly during conditions that involve server disconnections due to 
timeouts (very high disk load). Specifically, I've noticed that my 
.viminfo file got corrupted for the 3rd time today. Since this is root's 
.viminfo, and I'm running glfs as root, I don't have the logs to verify 
the disconnections, though. From what I can tell, a chunk of a dll 
somehow ends up in .viminfo, but I'm not sure which one.

On a different volume, I'm seeing other weirdness under the same high 
disk load conditions (software RAID check/resync on all server nodes). 
This seems to be specifically related to using writebehind+iocache on 
the client-side on one of he servers, exported via unfsd (the one from 
the gluster ftp site). What happens is that the /home volume simply 
seems to disappear underneath unfsd! The attached log indicates a 
glusterfsd crash.

This doesn't happen if I remove the writebehind and io-cache translators.

Other notable things about the setup that might help figure out the 
cause of this:

- The other two servers are idle - they are not serving any requests. 
They are, however, also under the same high disk load.

- writebehind and io-cache is only applied on one server the one behing 
used to export via unfsd. The other servers do not have those 
translators applied. The volume config is attached. It is called 
home-cache.vol, but this is the same file the log file refers to even 
though it is listed there as home.vol.

The problem specifically occurs when servers are undergoing high load of 
the described nature that causes disk latencies to go up massively. I 
have not observed any instances of a similar crash happening without the 
writebehind and io-cache translators.

Gordan

================================================================================
Version      : glusterfs 2.0.9 built on Dec 24 2009 23:13:15
git: v2.0.9
Starting Time: 2010-01-05 11:18:37
Command line : /usr/sbin/glusterfs --log-level=NORMAL --disable-direct-io-mode 
--volfile=/etc/glusterfs/home.vol /home 
PID          : 14917
System name  : Linux
Nodename     : raiden.shatteredsilicon.net
Kernel Release : 2.6.18-164.9.1.el5
Hardware Identifier: x86_64

Given volfile:
+------------------------------------------------------------------------------+
  1: volume home1
  2:    type protocol/client
  3:    option transport-type socket
  4:    option transport.address-family inet
  5:    option remote-host 10.2.0.11
  6:    option remote-port 6997
  7:    option remote-subvolume home1
  8: end-volume
  9: 
 10: volume home1-writebehind
 11:    type performance/write-behind
 12:    option cache-size 2MB           # default is equal to aggregate-size
 13:    option flush-behind on          # default is 'off'
 14:    option enable-O_SYNC on
 15:    subvolumes home1
 16: end-volume
 17: 
 18: volume home1-iocache
 19:    type performance/io-cache
 20:    option cache-size 64MB
 21:    option cache-timeout 2          # default is 1 second
 22:    subvolumes home1-writebehind
 23: end-volume
 24: 
 25: 
##############################################################################
 26: 
 27: volume home3
 28:    type protocol/client
 29:    option transport-type socket
 30:    option transport.address-family inet
 31:    option remote-host 10.2.0.13
 32:    option remote-port 6997
 33:    option remote-subvolume home3
 34: end-volume
 35: 
 36: volume home3-writebehind
 37:    type performance/write-behind
 38:    option cache-size 2MB           # default is equal to aggregate-size
 39:    option flush-behind on          # default is 'off'
 40:    option enable-O_SYNC on
 41:    subvolumes home3
 42: end-volume
 43: 
 44: volume home3-iocache
 45:    type performance/io-cache
 46:    option cache-size 64MB
 47:    option cache-timeout 2          # default is 1 second
 48:    subvolumes home3-writebehind
 49: end-volume
 50: 
 51: 
##############################################################################
 52: 
 53: volume home-store
 54:    type storage/posix
 55:    option directory /gluster/home
 56: end-volume
 57: 
 58: volume home2
 59:    type features/posix-locks
 60:    subvolumes home-store
 61: end-volume
 62: 
 63: volume home2-writebehind
 64:    type performance/write-behind
 65:    option cache-size 2MB           # default is equal to aggregate-size
 66:    option flush-behind on          # default is 'off'
 67:    option enable-O_SYNC on
 68:    subvolumes home2
 69: end-volume
 70: 
 71: volume home2-iocache
 72:    type performance/io-cache
 73:    option cache-size 64MB
 74:    option cache-timeout 2          # default is 1 second
 75:    subvolumes home2-writebehind
 76: end-volume
 77: 
 78: 
##############################################################################
 79: 
 80: volume server
 81:    type protocol/server
 82:    option transport-type socket
 83:    option transport.address-family inet
 84:    option transport.socket.listen-port 6997
 85:    subvolumes home2
 86:    option auth.addr.home2.allow 127.0.0.1,10.*
 87: end-volume
 88: 
 89: volume home
 90:    type cluster/replicate
 91:    subvolumes home2-iocache home1-iocache home3-iocache
 92:    option read-subvolume home2-iocache
 93:    option favorite-child home2-iocache
 94: end-volume

+------------------------------------------------------------------------------+
[2010-01-05 11:18:37] W [afr.c:2436:init] home: You have specified subvolume 
'home2-iocache' as the 'favorite child'. This means that if a discrepancy in 
the content or attributes (ownership, permission, etc.) of a file is detected 
among the subvolumes, the file on 'home2-iocache' will be considered the 
definitive version and its contents will OVERWRITE the contents of the file on 
other subvolumes. All versions of the file except that on 'home2-iocache' WILL 
BE LOST.
[2010-01-05 11:18:37] N [afr.c:2194:notify] home: Subvolume 'home2-iocache' 
came back up; going online.
[2010-01-05 11:18:37] N [afr.c:2194:notify] home: Subvolume 'home2-iocache' 
came back up; going online.
[2010-01-05 11:18:37] N [afr.c:2194:notify] home: Subvolume 'home2-iocache' 
came back up; going online.
[2010-01-05 11:18:37] N [glusterfsd.c:1306:main] glusterfs: Successfully started
[2010-01-05 11:18:37] N [client-protocol.c:5733:client_setvolume_cbk] home1: 
Connected to 10.2.0.11:6997, attached to remote volume 'home1'.
[2010-01-05 11:18:37] N [client-protocol.c:5733:client_setvolume_cbk] home1: 
Connected to 10.2.0.11:6997, attached to remote volume 'home1'.
[2010-01-05 11:18:37] N [client-protocol.c:5733:client_setvolume_cbk] home3: 
Connected to 10.2.0.13:6997, attached to remote volume 'home3'.
[2010-01-05 11:18:37] N [client-protocol.c:5733:client_setvolume_cbk] home3: 
Connected to 10.2.0.13:6997, attached to remote volume 'home3'.
[2010-01-05 11:18:37] N [server-protocol.c:7065:mop_setvolume] server: accepted 
client from 10.2.0.13:1015
[2010-01-05 11:18:37] N [server-protocol.c:7065:mop_setvolume] server: accepted 
client from 10.2.0.13:1014
[2010-01-05 11:18:47] N [server-protocol.c:7065:mop_setvolume] server: accepted 
client from 10.2.0.11:1009
[2010-01-05 11:18:47] N [server-protocol.c:7065:mop_setvolume] server: accepted 
client from 10.2.0.11:1008
[2010-01-05 11:21:05] E [posix.c:3156:do_xattrop] home-store: getxattr failed 
on /gordan/.gconf/apps/bluetooth-manager while doing xattrop: No such file or 
directory
pending frames:
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)
frame : type(1) op(LOOKUP)

patchset: v2.0.9
signal received: 11
time of crash: 2010-01-05 11:21:06
configuration details:
argp 1
backtrace 1
db.h 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 2.0.9
/lib64/libc.so.6[0x3f55e302d0]
/lib64/libc.so.6(memcpy+0x15b)[0x3f55e7bf0b]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(__wb_collapse_write_bufs+0x105)[0x2ac6b23bc045]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_process_queue+0xa8)[0x2ac6b23bcc68]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_writev+0x373)[0x2ac6b23bf0e3]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_writev+0x123)[0x2ac6b25c5a13]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_cbk+0x1d6)[0x2ac6b2e2dea6]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_frame_return+0x240)[0x2ac6b25c7390]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_dispatch_requests+0x25b)[0x2ac6b25c4f4b]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_readv+0x1fa)[0x2ac6b25c679a]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_write+0xd7)[0x2ac6b2e2c1c7]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_write_iter+0x5d)[0x2ac6b2e2dc8d]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_write_cbk+0x91)[0x2ac6b2e2dfb1]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_writev_cbk+0x7a)[0x2ac6b25c5aca]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_stack_unwind+0x6a)[0x2ac6b23bbe8a]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_do_ops+0x2c)[0x2ac6b23bcb7c]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_process_queue+0xf4)[0x2ac6b23bccb4]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_writev+0x373)[0x2ac6b23bf0e3]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_writev+0x123)[0x2ac6b25c5a13]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_cbk+0x1d6)[0x2ac6b2e2dea6]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_frame_return+0x240)[0x2ac6b25c7390]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_dispatch_requests+0x25b)[0x2ac6b25c4f4b]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_readv+0x1fa)[0x2ac6b25c679a]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_write+0xd7)[0x2ac6b2e2c1c7]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_read_write_iter+0x5d)[0x2ac6b2e2dc8d]
/usr/lib64/glusterfs/2.0.9/xlator/cluster/replicate.so(afr_sh_data_write_cbk+0x91)[0x2ac6b2e2dfb1]
/usr/lib64/glusterfs/2.0.9/xlator/performance/io-cache.so(ioc_writev_cbk+0x7a)[0x2ac6b25c5aca]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_stack_unwind+0x6a)[0x2ac6b23bbe8a]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_do_ops+0x2c)[0x2ac6b23bcb7c]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_process_queue+0xf4)[0x2ac6b23bccb4]
/usr/lib64/glusterfs/2.0.9/xlator/performance/write-behind.so(wb_sync_cbk+0xc5)[0x2ac6b23be2b5]
/usr/lib64/glusterfs/2.0.9/xlator/protocol/client.so(client_write_cbk+0x14e)[0x2ac6b21b13de]
/usr/lib64/glusterfs/2.0.9/xlator/protocol/client.so(protocol_client_pollin+0xca)[0x2ac6b21a18aa]
/usr/lib64/glusterfs/2.0.9/xlator/protocol/client.so(notify+0x212)[0x2ac6b21a84e2]
/usr/lib64/glusterfs/2.0.9/transport/socket.so(socket_event_handler+0xd3)[0x2aaaaaaafe33]
/usr/lib64/libglusterfs.so.0[0x3f56a27115]
/usr/sbin/glusterfs(main+0xa06)[0x403e96]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3f55e1d994]
/usr/sbin/glusterfs[0x402509]
---------

volume home1
        type protocol/client
        option transport-type socket
        option transport.address-family inet
        option remote-host 10.2.0.11
        option remote-port 6997
        option remote-subvolume home1
end-volume

volume home1-writebehind
        type performance/write-behind
        option cache-size 2MB           # default is equal to aggregate-size
        option flush-behind on          # default is 'off'
        option enable-O_SYNC on
        subvolumes home1
end-volume

volume home1-iocache
        type performance/io-cache
        option cache-size 64MB
        option cache-timeout 2          # default is 1 second
        subvolumes home1-writebehind
end-volume

##############################################################################

volume home3
        type protocol/client
        option transport-type socket
        option transport.address-family inet
        option remote-host 10.2.0.13
        option remote-port 6997
        option remote-subvolume home3
end-volume

volume home3-writebehind
        type performance/write-behind
        option cache-size 2MB           # default is equal to aggregate-size
        option flush-behind on          # default is 'off'
        option enable-O_SYNC on
        subvolumes home3
end-volume

volume home3-iocache
        type performance/io-cache
        option cache-size 64MB
        option cache-timeout 2          # default is 1 second
        subvolumes home3-writebehind
end-volume

##############################################################################

volume home-store
        type storage/posix
        option directory /gluster/home
end-volume

volume home2
        type features/posix-locks
        subvolumes home-store
end-volume

volume home2-writebehind
        type performance/write-behind
        option cache-size 2MB           # default is equal to aggregate-size
        option flush-behind on          # default is 'off'
        option enable-O_SYNC on
        subvolumes home2
end-volume

volume home2-iocache
        type performance/io-cache
        option cache-size 64MB
        option cache-timeout 2          # default is 1 second
        subvolumes home2-writebehind
end-volume

##############################################################################

volume server
        type protocol/server
        option transport-type socket
        option transport.address-family inet
        option transport.socket.listen-port 6997
        subvolumes home2
        option auth.addr.home2.allow 127.0.0.1,10.*
end-volume

volume home
        type cluster/replicate
        subvolumes home2-iocache home1-iocache home3-iocache
        option read-subvolume home2-iocache
        option favorite-child home2-iocache
end-volume