[Gluster-devel] Seg 11 with mainline-2.5 patch 185

[Dale Dude]

Linux (ubuntu dapper) 2.6.15. Fuse 2.6.5. Glusterfs mainline-2.5 patch 185.

glusterfs crashing with signal 11 on 'ls'. Was working for a few hours 
and now it crashes on any 'ls' of the mount. Odd thing, restarting 
glusterfsd and remounting fresh and restarting rsync it seems to be 
working ok (no errors). Running 6 rsyncs in parallel. As long as I dont 
do an 'ls' on the mount.

Things I did (mostly in sequence) leading up to this:
  * I was rsyncing for a few hours (rsync'd 75G).
  * Had aborted and restarted rsync a number of times.
  * Had rebooted once.
After one of these it started crashing. Dunno which cuz they were all 
done about the same time:
  * Tweaked network in sysctl.conf. A number of thing. Reset them back 
and reboot but still crashing.
  * Ran bonnie++ and aborted it.
  * Added write-behind xlator to glusterfs-client.vol. Removed when it 
wouldnt work with 'notify missing' error.

========================================
GDB glusterfs -c /core.5194:
Core was generated by 
`[glusterfs]                                                       '.
Program terminated with signal 11, Segmentation fault.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libglusterfs.so.0...done.
Loaded symbols for /lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libfuse.so.2...done.
Loaded symbols for /usr/local/lib/libfuse.so.2
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/tls/i686/cmov/librt.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/librt.so.1
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/transport/tcp/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/transport/tcp/client.so
Reading symbols from /lib/glusterfs/1.3.0-pre5/scheduler/rr.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/scheduler/rr.so
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0, 
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
1442            ret = value_data->len;
(gdb) bt
#0  fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0, 
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
#1  0xb76564f7 in unify_getxattr_cbk (frame=0xa, cookie=0x8094190, 
this=0x80570b8, op_ret=0, op_errno=0, value=0x0) at unify.c:3287
#2  0xb7663005 in client_getxattr_cbk (frame=0x8094190, args=0x8094ea8) 
at client-protocol.c:3431
#3  0xb7664f4c in notify (this=0x8056e08, event=2, data=0x808e6a0) at 
client-protocol.c:4126
#4  0xb7fd8982 in transport_notify (this=0x8094164, event=1) at 
transport.c:152
#5  0xb7fd90fe in sys_epoll_iteration (ctx=0x0) at epoll.c:54
#6  0xb7fd8b2d in poll_iteration (ctx=0x0) at transport.c:260
#7  0x0804a329 in main (argc=3, argv=0xbfdf9c84) at glusterfs.c:341

========================================
GLUSTERFSD.LOG:
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport] 
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp: 
clusterfs: connection disconnected
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport] 
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp: 
clusterfs: connection disconnected

========================================
GLUSTERFS.LOG:
2007-06-21 14:09:47 C [common-utils.c:205:gf_print_trace] 
debug-backtrace: Got signal (11), printing backtrace
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xb7f97499]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [0xffffe420]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so 
[0xb76164f7]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so 
[0xb7623005]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: 
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so(notify+0x855) 
[0xb7624f4c]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(transport_notify+0x37) [0xb7f98982]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(sys_epoll_iteration+0xd7) 
[0xb7f990fe]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7f98b2d]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [glusterfs] [0x804a329]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xd2) 
[0xb7e49ea2]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [glusterfs] [0x8049c71]

========================================
glusterfs-client.vol:
volume server1
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host 127.0.0.1     # IP address of the remote brick
        option remote-subvolume volumenamespace
end-volume

volume server1vol1
        type protocol/client
        option transport-type tcp/client     # for TCP/IP transport
        option remote-host 127.0.0.1     # IP address of the remote brick
        option remote-subvolume clusterfs1
end-volume

###################

volume bricks
 type cluster/unify
 option namespace server1
 option readdir-force-success on  # ignore failed mounts
 subvolumes server1vol1
 option scheduler rr
 option rr.limits.min-free-disk 5 #%
end-volume


========================================
glusterfs-server.vol:
volume clusterfs1
 type storage/posix
 option directory /volume1
end-volume

#######

volume volumenamespace
 type storage/posix
 option directory /volume.namespace
end-volume

###

volume clusterfs
 type protocol/server
 option transport-type tcp/server
 subvolumes clusterfs1 volumenamespace
 option auth.ip.clusterfs1.allow *
 option auth.ip.volumenamespace.allow *
end-volume