Re: [Libunwind-devel] UNW_EINVAL stepping past _L_lock_686 in pthread_mutex_lock

[Arun Sharma]

[ trimmed cc list, +milian, +tim ]

I looked into this a bit more today. _L_lock_9xx has hand coded unwind
info. dwarf_step() is able to decode it properly and determine that
the next frame is at

#2  0x00007ffff79a4fab in __GI___pthread_mutex_lock (mutex=0x60a360
<lock>) at pthread_mutex_lock.c:64

based on the debug log below:

    >_ULx86_64_reuse_frame: reuse frame ip=0x7fb9655e315c
cfa=0x7fb9647ede00 format=0 addr=0x0 offset=+0
              >_ULx86_64_dwarf_eval_expr: len=14, pushing cfa=0x7fb9647ede00
               >_ULx86_64_dwarf_eval_expr: OP_breg(r16,0x6)
               >_ULx86_64_dwarf_eval_expr: OP_const4s(-39810)
               >_ULx86_64_dwarf_eval_expr: OP_minus
               >_ULx86_64_dwarf_eval_expr: OP_const4s(-40249)
               >_ULx86_64_dwarf_eval_expr: OP_plus
              >_ULx86_64_dwarf_eval_expr: final value = 0x7fb9655e2fab
               >_ULx86_64_dwarf_step: returning 1
  >_ULx86_64_step: rbp null: 0
  >_ULx86_64_step: ret addr null: 1
  >_ULx86_64_step: returning 0

However, _ULx86_64_step determines that IP address column is null and
decides to terminate unwind. Gparser.c:719 (DWARF_MEM_LOC) always
evaluates to DWARF_NULL_LOC.

Support for DWARF_VAL_LOC() that Tim sent me in Feb works only for the
remote unwinding case. To make it work for the local unwinding case,
we'll need to change this code:

dwarf_config.h:

typedef struct dwarf_loc
  {
    unw_word_t val;
#ifndef UNW_LOCAL_ONLY
    unw_word_t type;            /* see X86_LOC_TYPE_* macros.  */
#endif
  }
dwarf_loc_t;

That is we add "type" information even for the local unwinding case to
distinguish between a value expression and an expression that involves
accessing memory.

I can probably whip up a patch that works, but we'll need to think
through the performance impact of increasing the size of dwarf_loc_t.

 -Arun


On Sat, Sep 27, 2014 at 1:34 PM, Jared Cantwell
<address@hidden> wrote:
> I pulled the tip of libunwind and while my test passes (libunwind
> thinks it walked the whole stack), the results are incorrect.  The
> second stack walked incorrectly stops at _L_lock_909 without an error,
> whereas with libunwind 1.1 it stops with an error.  The second stack
> should continue through to pthread_mutex_lock and all the way down to
> start_thread and clone.  I've compiled gdb with additional logging in
> stack walking and am in the process of comparing to the libunwind log
> info.
>
> --- not in pthread_mutex_lock ---
> thread_start(void*)
> start_thread
> clone
> ---------------------------------
> --- IN pthread_mutex_lock ---
> sig_handler(int)
> killpg
> __lll_lock_wait
> _L_lock_909
> ---------------------------------
> PASS: Full stack was walked.
>
> ~Jared
>
> On Sat, Sep 27, 2014 at 10:58 AM, Arun Sharma <address@hidden> wrote:
>> On Fri, Sep 26, 2014 at 2:15 PM, Jared Cantwell
>> <address@hidden> wrote:
>>> I have reproduced this on ubuntu 14.04 and 13.10.  Configuration
>>> information below.  I've also tested on 3.14 kernel and eglibc 2.18
>>> without any luck.  Arun, if my test passes for you on 13.10 can you
>>> provide information on your setup so I can compare?
>>
>> ubuntu 13.10
>> 3.11.0-12-generic
>> eglibc 2.17-93ubuntu4
>> g++ 4.8.1-10ubuntu8
>>
>> I can't tell which version of libunwind you tested. 1.1 is likely to
>> have bugs that have since been fixed.
>>
>>>
>>> Does the fact that gdb can unwind this stack mean anything?  Like that
>>> the unwind information is in libpthread in some form that libunwind
>>> doesn't quite understand?
>>
>> This strongly suggests the unwind info is there, but your version of
>> libunwind is unable to find it.
>>
>> There is also a possibility that gdb has a heuristic that works in the
>> absence of unwind info, but libunwind's heuristic doesn't work.
>>
>> Steps I used to test:
>>
>> cd libunwind; // commit id 781d5d5
>> make DESTDIR=/tmp/foo install
>> g++ unwind_repro.cpp -o unwind_repro
>> /tmp/foo/usr/local/lib/libunwind.a -llzma -lpthread
>> ./unwind_repro // ran it 10 times
>>
>>  -Arun