Re: [Gluster-devel] gluster server 3.0 segfaults on ibverbs setup

[Shehjar Tikoo]

Hi

Are you by any chance running the client and server from
two different releases say, a client from 2.x versions and
a server from 3.x version or vice-versa?

-Shehjar

Mickey Mazarick wrote:
> Kernel is 2.6.18.6 ofed 1.4 for infiniband
> No errors no warnings.. the log file ends with:
> +------------------------------------------------------------------------------+ 
>
> [2009-12-09 07:54:56] N [glusterfsd.c:1361:main] glusterfs: Successfully 
> started
> pending frames:
>
> I didn't know it segfaulted until I ran it at a local process.
>
>
> The vol file is:
>
> volume br1
> type storage/posix
> option directory /mnt/gluster/system
> end-volume
>
> volume brick
>  type features/posix-locks
>  subvolumes br1
> end-volume
>
> volume main1s
> type storage/posix
> option directory /mnt/gluster/main1/export1
> end-volume
>
> volume main1a
>  type features/posix-locks
>  subvolumes main1s
> end-volume
>
> volume main1
>  type performance/io-threads
>  option thread-count 2  # deault is 1
>  subvolumes main1a
> end-volume
>
> volume main2s
> type storage/posix
> option directory /mnt/gluster/main2
> end-volume
>
> volume main2
>  type features/posix-locks
> #  option mandatory on
>  subvolumes main2s
> end-volume
>
> volume server
> type protocol/server
> option transport-type ib-verbs
> subvolumes brick main2 main1
> option auth.addr.main1.allow *
> option auth.addr.main2.allow *
> option auth.addr.brick.allow *
> end-volume
>
>
>
> Core file results:
>
> warning: core file may not match specified executable file.
> Core was generated by `/usr/local/sbin/glusterfsd -p 
> /var/run/glusterfs-server.vol.ibverbs.pid -f /etc'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
> Loaded symbols for /usr/local/lib/libglusterfs.so.0
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/tls/libpthread.so.0...done.
> Loaded symbols for /lib64/tls/libpthread.so.0
> Reading symbols from /lib64/tls/libc.so.6...done.
> Loaded symbols for /lib64/tls/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from 
> /usr/local/lib/glusterfs/3.0.0/xlator/storage/posix.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/3.0.0/xlator/storage/posix.so
> Reading symbols from 
> /usr/local/lib/glusterfs/3.0.0/xlator/features/posix-locks.so...done.
> Loaded symbols for 
> /usr/local/lib/glusterfs/3.0.0/xlator/features/posix-locks.so
> Reading symbols from 
> /usr/local/lib/glusterfs/3.0.0/xlator/performance/io-threads.so...done.
> Loaded symbols for 
> /usr/local/lib/glusterfs/3.0.0/xlator/performance/io-threads.so
> Reading symbols from 
> /usr/local/lib/glusterfs/3.0.0/xlator/protocol/server.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/3.0.0/xlator/protocol/server.so
> Reading symbols from 
> /usr/local/lib/glusterfs/3.0.0/transport/ib-verbs.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/3.0.0/transport/ib-verbs.so
> Reading symbols from /usr/lib64/libibverbs.so.1...done.
> Loaded symbols for /usr/lib64/libibverbs.so.1
> Reading symbols from /usr/lib64/libmthca-rdmav2.so...done.
> Loaded symbols for /usr/lib64/libmthca-rdmav2.so
> Reading symbols from /usr/lib64/libcxgb3-rdmav2.so...done.
> Loaded symbols for /usr/lib64/libcxgb3-rdmav2.so
> Reading symbols from /usr/lib64/libmlx4-rdmav2.so...done.
> Loaded symbols for /usr/lib64/libmlx4-rdmav2.so
> Reading symbols from /usr/lib64/libnes-rdmav2.so...done.
> Loaded symbols for /usr/lib64/libnes-rdmav2.so
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /usr/local/lib/glusterfs/3.0.0/auth/addr.so...done.
> Loaded symbols for /usr/local/lib/glusterfs/3.0.0/auth/addr.so
> #0  0x00002abdc82d5427 in gf_print_trace (signum=11) at common-utils.c:413
> 413
> (gdb) backtrace
> #0  0x00002abdc82d5427 in gf_print_trace (signum=11) at common-utils.c:413
> #1  <signal handler called>
> #2  0x00002abdc8fd0e9f in server_decode_groups (frame=0x522d58, 
> hdr=0x522b90) at server-protocol.c:6124
> #3  0x00002abdc8fd10a3 in get_frame_for_call (trans=0x520190, 
> hdr=0x522b90) at server-protocol.c:6158
> #4  0x00002abdc8fd132b in protocol_server_interpret (this=0x511890, 
> trans=0x520190, hdr_p=0x522b90 "", hdrlen=301, iobuf=0x0)
>    at server-protocol.c:6288
> #5  0x00002abdc8fd220b in protocol_server_pollin (this=0x511890, 
> trans=0x520190) at server-protocol.c:6681
> #6  0x00002abdc8fd23e0 in notify (this=0x511890, event=Variable "event" 
> is not available.
> ) at server-protocol.c:6737
> #7  0x00002abdc82cd984 in xlator_notify (xl=0x511890, event=2, 
> data=0x520190) at xlator.c:923
> #8  0x00002aaaaaaaf048 in ib_verbs_recv_completion_proc (data=Variable 
> "data" is not available.
> ) at ib-verbs.c:1192
> #9  0x00000033c1906137 in start_thread () from /lib64/tls/libpthread.so.0
> #10 0x00000033c0ac7113 in clone () from /lib64/tls/libc.so.6