4 daemons, 4 clients. My configs are in the "no element to prune"
thread on July 10.
I am running apache with html and php.
I also just noticed that the load on nodes 2 and 4 of my servers went
up to over 30 and I am unable to stop apache. I had to kill gluserfs
and glusterfsd on those two nodes.
I can provide you with a temporary remote login if that would help.
-pooya
On Jul 12, 2007, at 1:34 AM, Amar S. Tumballi wrote:
> Hi Pooya,
> let us know your setup.. ie, what application you were running and
> howmany clients.
>
> -amar
>
> On 7/12/07, Pooya Woodcock < address@hidden> wrote:Also on
> another node, I got this segfault. This one was first then I
> got the other one on node4 that I emailed eearlier.
>
> address@hidden ~]# gdb glusterfs -c /core.13358
> GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
> libthread_db library "/lib64/tls/libthread_db.so.1".
>
> Core was generated by `
> [glusterfs]
> '.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
> Loaded symbols for /usr/local/lib/libglusterfs.so.0
> Reading symbols from /usr/local/lib/libibverbs.so.1...done.
> Loaded symbols for /usr/local/lib/libibverbs.so.1
> Reading symbols from /usr/local/lib/libfuse.so.2...done.
> Loaded symbols for /usr/local/lib/libfuse.so.2
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/tls/libpthread.so.0...done.
> Loaded symbols for /lib64/tls/libpthread.so.0
> Reading symbols from /lib64/tls/libc.so.6...done.
> Loaded symbols for /lib64/tls/libc.so.6
> Reading symbols from /usr/lib64/libsysfs.so.1...done.
> Loaded symbols for /usr/lib64/libsysfs.so.1
> Reading symbols from /lib64/tls/librt.so.1...done.
> Loaded symbols for /lib64/tls/librt.so.1
> Reading symbols from /lib64/libselinux.so.1...done.
> Loaded symbols for /lib64/libselinux.so.1
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> protocol/client.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> protocol/client.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> cluster/afr.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> cluster/afr.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> cluster/unify.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> cluster/unify.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> performance/io-threads.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> performance/io-threads.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> performance/read-ahead.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> performance/read-ahead.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
> tcp/client.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
> tcp/client.so
> Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/scheduler/
> rr.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/scheduler/
> rr.so
> Reading symbols from /lib64/libgcc_s.so.1...done.
> Loaded symbols for /lib64/libgcc_s.so.1
> #0 unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
> flags=32770) at unify.c:956
> 956 loc_t tmp_loc = {
> (gdb) bt
> #0 unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
> flags=32770) at unify.c:956
> #1 0x0000002a95bc8121 in iot_open (frame=0x2a960d2190,
> this=0x515700, loc=0x2a960cd178, flags=32770) at io-threads.c:88
> #2 0x0000002a95ccc2a7 in ra_open (frame=0x2a960d1800, this=0x5160a0,
> loc=0x2a960cd178, flags=32770) at read-ahead.c:207
> #3 0x0000000000405898 in fuse_open (req=Variable "req" is not
> available.
> ) at fuse-bridge.c:1017
> #4 0x0000002a95783ffe in do_open (req=Variable "req" is not
> available.
> ) at fuse_lowlevel.c:586
> #5 0x0000000000407b52 in fuse_transport_notify (xl=Variable "xl" is
> not available.
> ) at fuse-bridge.c:1937
> #6 0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable "ctx" is
> not available.
> ) at epoll.c:54
> #7 0x0000000000402f4e in main (argc=7, argv=0x7fbffff9d8) at
> glusterfs.c:348
>
>
>
>
> On Jul 11, 2007, at 9:58 AM, Pooya Woodcock wrote:
>
> > Hi Avati,
> > Confirmed fixed. Thanks, running great so far.
> > Pooya
> >
> > On Jul 10, 2007, at 11:50 PM, Anand Avati wrote:
> >
> >> Pooya,
> >> this bug is fixed in patch 295. can you please confirm if some
> >> operations were performed when one (or more) server was down to
> >> reproduce this bug?
> >>
> >> thanks,
> >> avati
> >>
> >> 2007/7/11, Pooya Woodcock < address@hidden>:This is on
> >> patch 293 in 2.5 mainline.
> >> -Pooya
> >>
> >> On Jul 10, 2007, at 7:40 PM, Amar S. Tumballi wrote:
> >>
> >> > Hi Pooya,
> >> > Thanks for the bt, but can you tell us which version (or
> patch in
> >> > tla) are you using?
> >> >
> >> > -amar
> >> >
> >> > On 7/11/07, Pooya Woodcock < address@hidden> wrote:Here is
> >> > gdb output on the core..
> >> >
> >> > address@hidden ~]# gdb glusterfs -c /core.6612
> >> > GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
> >> > Copyright 2004 Free Software Foundation, Inc.
> >> > GDB is free software, covered by the GNU General Public License,
> >> and
> >> > you are
> >> > welcome to change it and/or distribute copies of it under certain
> >> > conditions.
> >> > Type "show copying" to see the conditions.
> >> > There is absolutely no warranty for GDB. Type "show warranty"
> for
> >> > details.
> >> > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
> >> > libthread_db library "/lib64/tls/libthread_db.so.1".
> >> >
> >> > Core was generated by `
> >> > [glusterfs]
> >> > '.
> >> > Program terminated with signal 11, Segmentation fault.
> >> > Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
> >> > Loaded symbols for /usr/local/lib/libglusterfs.so.0
> >> > Reading symbols from /usr/local/lib/libibverbs.so.1...done.
> >> > Loaded symbols for /usr/local/lib/libibverbs.so.1
> >> > Reading symbols from /usr/local/lib/libfuse.so.2...done.
> >> > Loaded symbols for /usr/local/lib/libfuse.so.2
> >> > Reading symbols from /lib64/libdl.so.2...done.
> >> > Loaded symbols for /lib64/libdl.so.2
> >> > Reading symbols from /lib64/tls/libpthread.so.0...done.
> >> > Loaded symbols for /lib64/tls/libpthread.so.0
> >> > Reading symbols from /lib64/tls/libc.so.6...done.
> >> > Loaded symbols for /lib64/tls/libc.so.6
> >> > Reading symbols from /usr/lib64/libsysfs.so.1...done.
> >> > Loaded symbols for /usr/lib64/libsysfs.so.1
> >> > Reading symbols from /lib64/tls/librt.so.1...done.
> >> > Loaded symbols for /lib64/tls/librt.so.1
> >> > Reading symbols from /lib64/libselinux.so.1...done.
> >> > Loaded symbols for /lib64/libselinux.so.1
> >> > Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> >> > Loaded symbols for /lib64/ld- linux-x86-64.so.2
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> xlator/
> >> > protocol/client.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> >> > protocol/client.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> xlator/
> >> > cluster/afr.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> >> > cluster/afr.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> xlator/
> >> > cluster/unify.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> >> > cluster/unify.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> xlator/
> >> > performance/io-threads.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> >> > performance/io- threads.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> xlator/
> >> > performance/read-ahead.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> >> > performance/read- ahead.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> >> transport/
> >> > tcp/client.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/
> transport/
> >> > tcp/client.so
> >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> >> scheduler/
> >> > rr.so...done.
> >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/
> scheduler/
> >> > rr.so
> >> > Reading symbols from /lib64/libgcc_s.so.1...done.
> >> > Loaded symbols for /lib64/libgcc_s.so.1
> >> > #0 inode_unref (inode=0x0) at inode.c:403
> >> > 403 inode_table_t *table = inode->table;
> >> >
> >> > (gdb) backtrace
> >> > #0 inode_unref (inode=0x0) at inode.c:403
> >> > #1 0x0000002a959ad6bc in afr_forget (frame=dwarf2_read_address:
> >> > Corrupted DWARF expression.
> >> > ) at afr.c:468
> >> > #2 0x0000002a95564a44 in __destroy_inode (inode=0x2a9619bc10) at
> >> > inode.c:250
> >> > #3 0x0000002a95564cfc in inode_unref (inode=0x2a9619bc10) at
> >> inode.c:
> >> > 413
> >> > #4 0x0000002a95abc289 in unify_forget (frame=Variable "frame"
> >> is not
> >> > available.
> >> > ) at unify.c:407
> >> > #5 0x0000002a95564a44 in __destroy_inode (inode=0x2a9619b8b0) at
> >> > inode.c:250
> >> > #6 0x0000002a95564cfc in inode_unref (inode=0x2a9619b8b0) at
> >> inode.c:
> >> > 413
> >> > #7 0x0000000000407370 in fuse_forget_notify (frame=Variable
> >> "frame"
> >> > is not available.
> >> > ) at fuse-bridge.c:1652
> >> > #8 0x0000002a95564a44 in __destroy_inode (inode=0x2a9608f0c0) at
> >> > inode.c:250
> >> > #9 0x0000002a95564cfc in inode_unref (inode=0x2a9608f0c0) at
> >> inode.c:
> >> > 413
> >> > #10 0x0000000000403993 in fuse_forget (req=0x2a9607de50,
> >> ino=Variable
> >> > "ino" is not available.
> >> > ) at fuse-bridge.c:363
> >> > #11 0x0000000000407b52 in fuse_transport_notify (xl=Variable
> >> "xl" is
> >> > not available.
> >> > ) at fuse-bridge.c:1937
> >> > #12 0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable
> >> "ctx" is
> >> > not available.
> >> > ) at epoll.c:54
> >> > #13 0x0000000000402f4e in main (argc=7, argv=0x7fbffffa08) at
> >> > glusterfs.c:348
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Jul 10, 2007, at 1:36 PM, Pooya Woodcock wrote:
> >> >
> >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:338:fuse_lookup]
> glusterfs-
> >> > > fuse: LOOKUP 6832132/sess_3c1f96159c610b755a7def44f036a11b (/
> >> > > php_sessions/sess_3c1f96159c610b755a
> >> > > 7def44f036a11b)
> >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] ns/inode:
> >> create
> >> > > inode(6832716)
> >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] ns/inode:
> >> > > activating inode(6832716), lru=0/1000
> >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] unify1/
> inode:
> >> > > create inode(6832716)
> >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] unify1/
> inode:
> >> > > activating inode(6832716), lru=0/1000
> >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] brick3-afr/
> >> > > inode: create inode(10535464)
> >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] brick3-afr/
> >> > > inode: activating inode(10535464), lru=0/1000
> >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] afr3/inode:
> >> > > create inode(10535464)
> >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] afr3/inode:
> >> > > activating inode(10535464), lru=0/1000
> >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:288:fuse_entry_cbk]
> >> glusterfs-
> >> > > fuse: ENTRY => 6832716
> >> > > 2007-07-10 13:30:21 D [ inode.c:511:__create_inode] fuse/inode:
> >> > > create inode(6832716)
> >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> >> > > activating inode(6832716), lru=454/1024
> >> > > 2007-07-10 13:30:21 D [ inode.c:332:__passive_inode] fuse/
> inode:
> >> > > passivating inode(6832716), lru=455/1024
> >> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
> >> > > passivating inode(6832132), lru=456/1024
> >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:775:fuse_unlink]
> glusterfs-
> >> > > fuse: UNLINK 6832132/sess_3c1f96159c610b755a7def44f036a11b
> >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> >> > > activating inode(6832716), lru=455/1024
> >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> >> > > activating inode(6832132), lru=454/1024
> >> > > 2007-07-10 13:30:21 D [inode.c :332:__passive_inode] fuse/
> inode:
> >> > > passivating inode(6832716), lru=455/1024
> >> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
> >> > > passivating inode(6832132), lru=456/1024
> >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] fuse/inode:
> >> > > activating inode(6832716), lru=455/1024
> >> > > 2007-07-10 13:30:21 C [common-utils.c:208:gf_print_trace]
> debug-
> >> > > backtrace: Got signal (11), printing backtrace
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(gf_print_trace
> +0x22)
> >> > > [0x2a95562782]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /lib64/tls/libc.so.6 [0x37fbc2e2b0]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x16)
> >> > > [0x2a95564bc6]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/glusterfs/1.3.0-pre5.2 /xlator/
> cluster/
> >> > > afr.so [0x2a959ad6bc]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c :210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> >> > > [0x2a95564cfc]
> >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> cluster/
> >> > > unify.so(unify_forget+0x29) [0x2a
> >> > > 95abc289]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> >> > > [0x2a95564cfc]
> >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> debug-
> >> > > backtrace: [glusterfs] [0x407370]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> >> > > [0x2a95564cfc]
> >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> debug-
> >> > > backtrace: [glusterfs] [0x403993]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: [glusterfs] [0x407b52]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration
> >> > > +0xc6) [0x2a95563e86]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: [glusterfs] [0x402f4e]
> >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: /lib64/tls/libc.so.6(__libc_start_main+0xdb)
> >> > [0x37fbc1c3fb]
> >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> debug-
> >> > > backtrace: [glusterfs] [0x40290a]
> >> > >
> >> > >
> >> > > _______________________________________________
> >> > > Gluster-devel mailing list
> >> > > address@hidden
> >> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Gluster-devel mailing list
> >> > address@hidden
> >> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >> >
> >> >
> >> >
> >> > --
> >> > Amar Tumballi
> >> > http://amar.80x25.org
> >> > [bulde on #gluster/irc.gnu.org]
> >>
> >>
> >>
> >> _______________________________________________
> >> Gluster-devel mailing list
> >> address@hidden
> >> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >>
> >>
> >>
> >> --
> >> Anand V. Avati
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
> --
> Amar Tumballi
> http://amar.80x25.org
> [bulde on #gluster/irc.gnu.org]
_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel