[Gluster-devel] libc6 reported errors in glusterfs logs

[Jordi Moles]

hi,

i've kept a testing mail service for days, with no other problem than 
some low performance under some particular situations.
But now, i'm starting to get problems with libc6.

I've got 4 postfix servers sharing a glusterfs system, version 
mainline-2.5, patch 690.

I've also got 6 node servers sharing the storage space that postfix 
servers use.

all nodes are still working, and also postfix1. The other ones can't 
access the glusterfs once this libc6 message has been logged in the 
glusterfs log file.

these are the logs from all the parts:

********postfix1 (it still works) ******

........
2008-03-07 08:49:39 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22189179: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:49:46 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22190414: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:49:53 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22191647: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:49:54 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22192847: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:49:55 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22194013: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:01 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22195227: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:03 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22196474: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:04 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22197676: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:05 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22198842: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:06 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22200100: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:07 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22201281: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:11 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22202518: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)
2008-03-07 08:50:12 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
22203686: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com 
=> -1 (116)

************************

******postfix2**********

2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797528: /dummymassiu/tmp/1204830430.P5800.postfix04gluster01.cdmon.com 
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797528: /dummymassiu/tmp/1204830430.P5800.postfix04gluster01.cdmon.com 
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797636: /dummymassiu/tmp/1204830431.P5798.postfix04gluster01.cdmon.com 
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797636: /dummymassiu/tmp/1204830431.P5798.postfix04gluster01.cdmon.com 
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797684: /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com 
=> -1 (2)
2008-03-06 19:08:09 E [unify.c:261:unify_lookup_cbk] ultim: Revalidate 
failed for /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
797686: /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com 
=> -1 (2)


TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 19:08:09
Signal Number : 11

/lib/libc.so.6[0x2b9e19944110]
/usr/lib/libglusterfs.so.0(inode_unref+0x16)[0x2b9e195f1a46]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeec4f2]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeec536]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaef0b0f]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaef0cd2]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup_cbk+0x2f2)[0x2aaaaade5872]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_sync_ownership_permission+0x473)[0x2aaaaabc7773]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_check_ctime_version+0x27c)[0x2aaaaabccdbc]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_lookup_cbk+0x501)[0x2aaaaabce321]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(client_lookup_cbk+0x1ed)[0x2aaaaaab84cd]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x8fc)[0x2aaaaaab18bc]
/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xbf)[0x2b9e195efdcf]
/usr/lib/libglusterfs.so.0(poll_iteration+0x75)[0x2b9e195ef2f5]
[glusterfs](main+0x64f)[0x40272f]
/lib/libc.so.6(__libc_start_main+0xda)[0x2b9e199314ca]
[glusterfs][0x401c9a]


***********************


********postfix3**************


2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
399453: /dummymassiu/tmp/1204828340.P2724.postfix01gluster01.cdmon.com 
=> -1 (2)
2008-03-06 18:33:23 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse: 
401349: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com 
=> 63434 Rehashing f000/0
2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
401353: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com 
=> -1 (2)
2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
401353: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com 
=> -1 (2)


TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 18:33:23
Signal Number : 11

/lib/libc.so.6[0x2b9e512a3110]
/lib/libc.so.6[0x2b9e512de7ff]
/lib/libc.so.6[0x2b9e512e0789]
/lib/libc.so.6(calloc+0x96)[0x2b9e512e1e46]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup+0x54)[0x2aaaaade2c14]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeef3f7]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeeface]
/lib/libpthread.so.0[0x2b9e51164f1a]
/lib/libc.so.6(__clone+0x72)[0x2b9e5133d5d2]


****************************


******postfix4**********

2008-03-06 20:54:15 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
2606669: /dummymassiu/tmp/1204836803.P11411.postfix01gluster01.cdmon.com 
=> -1 (2)
2008-03-06 20:54:15 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse: 
2606669: /dummymassiu/tmp/1204836803.P11411.postfix01gluster01.cdmon.com 
=> -1 (2)
2008-03-06 20:54:15 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse: 
2606806: /dummymassiu/tmp/1204836832.P10986.postfix01gluster01.cdmon.com 
=> 103147 Rehashing 9000/0
2008-03-06 20:54:15 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse: 
2606807: /dummymassiu/tmp/1204836832.P10986.postfix01gluster01.cdmon.com 
=> 103147 Rehashing 9000/0


TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 20:54:15
Signal Number : 11

/lib/libc.so.6[0x2b6377f4e110]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup_cbk+0x2c)[0x2aaaaade55ac]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_lookup_cbk+0x2cc)[0x2aaaaabce0ec]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(client_lookup_cbk+0x1ed)[0x2aaaaaab84cd]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x8fc)[0x2aaaaaab18bc]
/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xbf)[0x2b6377bf9dcf]
/usr/lib/libglusterfs.so.0(poll_iteration+0x75)[0x2b6377bf92f5]
[glusterfs](main+0x64f)[0x40272f]
/lib/libc.so.6(__libc_start_main+0xda)[0x2b6377f3b4ca]
[glusterfs][0x401c9a]


**********************


*******node1*********

2008-03-06 16:54:15 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 17:01:38 E [protocol.c:259:gf_block_unserialize_transport] 
ultim: EOF from peer (192.168.1.219:1021)
2008-03-06 17:01:38 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 17:01:38 E [protocol.c:259:gf_block_unserialize_transport] 
ultim: EOF from peer (192.168.1.219:1023)
2008-03-06 17:01:38 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 18:33:22 E [protocol.c:259:gf_block_unserialize_transport] 
ultim: EOF from peer (192.168.1.34:1019)
2008-03-06 18:33:22 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 18:33:22 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim: 
transport_writev failed
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim: 
transport_writev failed
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim: 
transport_writev failed
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected

***********************

***********node2***********

2008-03-06 17:01:37 E [protocol.c:259:gf_block_unserialize_transport] 
ultim: EOF from peer (192.168.1.219:1022)
2008-03-06 17:01:37 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 17:32:45 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: 20688!
2008-03-06 17:32:45 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: 20688!
2008-03-06 17:33:21 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: 2080!
2008-03-06 17:33:21 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: 2080!
2008-03-06 18:04:41 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: -27456!
2008-03-06 18:04:41 E [server-protocol.c:4862:server_lk] espai: Unknown 
lock type: -27456!
2008-03-06 18:33:23 E [protocol.c:259:gf_block_unserialize_transport] 
ultim: EOF from peer (192.168.1.34:1018)
2008-03-06 18:33:23 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 18:33:23 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection 
disconnected


***************************