Re: [Gluster-devel] Application segfaults when executed on glusterfs mounted volume

[Erik Osterman]

Unfortunately, our mapserv isn't compiled with debugging symbols. If it 
comes down to it, I can try and arrange that.


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209813296 (LWP 14678)]
0x080c54e3 in __libc_csu_init ()
(gdb) bt
#0  0x080c54e3 in __libc_csu_init ()
#1  0x0804caa1 in _start ()
(gdb)


... And the tail end of the strace.

open("/usr/lib/libk5crypto.so.3", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220W\245"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=157196, ...}) = 0
mmap2(0x42a52000, 155040, PROT_READ|PROT_EXEC, 
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x42a52000
mmap2(0x42a77000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25) = 0x42a77000
close(3)                                = 0
open("/lib/libresolv.so.2", O_RDONLY)   = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\241"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=80696, ...}) = 0
mmap2(0x42808000, 80072, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
3, 0) = 0x42808000
mmap2(0x42818000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf) = 0x42818000
mmap2(0x4281a000, 6344, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4281a000
close(3)                                = 0
open("/usr/lib/libkrb5support.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \266\233"..., 
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=31236, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e76000
mmap2(0x429ba000, 32488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
3, 0) = 0x429ba000
mmap2(0x429c1000, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0x429c1000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e75000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0xb7e74000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7e746d0, 
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, 
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x42818000, 4096, PROT_READ)   = 0
mprotect(0x427ba000, 4096, PROT_READ)   = 0
mprotect(0xb7e7d000, 4096, PROT_READ)   = 0
mprotect(0x42762000, 4096, PROT_READ)   = 0
mprotect(0x42c2b000, 12288, PROT_READ)  = 0
mprotect(0xb7ecf000, 4096, PROT_READ)   = 0
mprotect(0x42757000, 8192, PROT_READ)   = 0
mprotect(0x42605000, 4096, PROT_READ)   = 0
munmap(0xb7f5e000, 36223)               = 0
set_tid_address(0xb7e74718)             = 14686
set_robust_list(0xb7e74720, 0xc)        = -1 ENOSYS (Function not 
implemented)
rt_sigaction(SIGRTMIN, {0x427aa2c0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x427aa340, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="domU-12-31-36-00-35-03", ...}) = 0
futex(0x42c314fc, FUTEX_WAKE, 2147483647) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 14686 detached




Anand Avati wrote:
> Erik,
>  please give a 'bt' command output from gdb, also if possible an
> strace log of running mapsrv over glusterfs (to see the last FS
> operation or more clues)
>
>
> regards,
> avati
>
> On Tue, May 01, 2007 at 05:32:12PM -0700, Erik Osterman wrote:
>   
> > If I try to run our mapserv application from a glusterfs mounted volume 
> > I consistently get segfaults. Yet if I copy that binary from the 
> > glusterfs volume to /tmp, it works fine.
> >
> > I only have the postfix translator enabled on the glusterfsd processes.  
> > The clients are are running afr (*:2) between 3 bricks, with unify, 
> > writebehind, readhead enabled. stat-prefetch and io-threads are both 
> > disabled.
> >
> > Where would be a good place to start diagnosing this problem?
> >
> >
> > Thanks,
> >
> > Erik Osterman
> >
> > -- Transcript --
> >
> > # ./mapserv
> > Segmentation fault
> >
> > GDB reports,
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread -1209043248 (LWP 11563)]
> > 0x080c54e3 in __libc_csu_init ()
> >
> > # cp mapserv /tmp
> > # /tmp/mapserv
> > This script can only be used to decode form results and
> > should be initiated as a CGI process via a httpd server.
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >     
>
>