Re: MIPS unwind only backtraces 2 local levels...

Greetings,

Thanks again for your detailed responses.

Our system is indeed, MIPS + MUSL, and we rebuilt it with the options you provided. I am still at a loss as to why we only get 2 levels.

I am attempting to overload "malloc" and get backtraces for when certain threads allocate memory. The interesting thing is that the 2 levels i am getting are "malloc" and my internal malloc which are located in the same file. I am not using LD_PRELOAD to do this, so the linked image has the internal malloc and not the MUSL one.

Since this is an embedded linux system, i cant really run the image with full dwarf info, but rather a stripped one. I confirmed that the .eh_frame section is there, and not stripped.

I enabled and set the UNW_DEBUG_LEVEL, the output is below.

It appears to find the eh_frame data (i think), searches and finds on CIE, then stops. But i dont see an indication as to why it stopped.

Is there anything you can discern from this trace that offers any clues ?

Thanks Again,

-Lenny

My backtrace test print code :

=========================================================

#define BT_ARRAY_SIZE (10)

void *bt[BT_ARRAY_SIZE];

int Count;

memset(&bt, 0, sizeof(bt));

Count = unw_backtrace((void **)&bt, BT_ARRAY_SIZE);

if(Count)

{

fprintf(stdout, "[ALLOC] Alloc : %d bytes : %d : %p,%p,%p,%p,%p,%p,%p \n",

Size, Count, bt[0], bt[1], bt[2], bt[3], bt[4], bt[5], bt[6]);

}

My image shows the following sections :

=======================================================

There are 32 section headers, starting at offset 0x12ecbe8:

Section Headers:

[Nr] Name Type Addr Off Size ES Flg Lk Inf Al

[ 0] NULL 00000000 000000 000000 00 0 0 0

[ 1] .interp PROGBITS 00000174 000174 00000d 00 A 0 0 1

[ 2] .MIPS.abiflags MIPS_ABIFLAGS 00000188 000188 000018 18 A 0 0 8

[ 3] .note.gnu.build-i NOTE 000001a0 0001a0 000024 00 A 0 0 4

[ 4] .dynamic DYNAMIC 000001c4 0001c4 000158 08 A 7 0 4

[ 5] .hash HASH 0000031c 00031c 016da8 04 A 6 0 4

[ 6] .dynsym DYNSYM 000170c4 0170c4 0476d0 10 A 7 2 4

[ 7] .dynstr STRTAB 0005e794 05e794 1bf709 00 A 0 0 1

[ 8] .gnu.version VERSYM 0021de9e 21de9e 008eda 02 A 6 0 2

[ 9] .gnu.version_r VERNEED 00226d78 226d78 0001b0 00 A 7 3 4

[10] .rel.dyn REL 00226f28 226f28 031880 08 A 6 0 4

[11] .init PROGBITS 00295b40 295b40 00001c 00 AX 0 0 4

[12] .text PROGBITS 00295b60 295b60 dea060 00 AX 0 0 16

[13] .MIPS.stubs PROGBITS 0107fbc0 107fbc0 001680 00 AX 0 0 4

[14] .fini PROGBITS 01081240 1081240 00001c 00 AX 0 0 4

[15] .rodata PROGBITS 01081260 1081260 10d1f0 00 A 0 0 16

[16] .eh_frame_hdr PROGBITS 0118e450 118e450 017ae4 00 A 0 0 4

[17] .eh_frame PROGBITS 011b6000 11a6000 0da0e0 00 WA 0 0 4

[18] .gcc_except_table PROGBITS 012900e0 12800e0 03383f 00 WA 0 0 4

[19] .init_array INIT_ARRAY 012c3920 12b3920 000bec 04 WA 0 0 4

[20] .fini_array FINI_ARRAY 012c450c 12b450c 000004 04 WA 0 0 4

[21] .data.rel.ro PROGBITS 012c4510 12b4510 021bbc 00 WA 0 0 8

[22] .data PROGBITS 012e60d0 12d60d0 002860 00 WA 0 0 16

[23] .rld_map PROGBITS 012e8930 12d8930 000004 00 WA 0 0 4

[24] .got PROGBITS 012e8940 12d8940 014148 04 WAp 0 0 16

[25] .sdata PROGBITS 012fca88 12eca88 000004 00 WAp 0 0 4

[26] .sbss NOBITS 012fca8c 12eca8c 000014 00 WAp 0 0 4

[27] .bss NOBITS 012fcaa0 12eca8c 0dc0ac 00 WA 0 0 16

[28] .comment PROGBITS 00000000 12eca8c 000011 01 MS 0 0 1

[29] .gnu.attributes GNU_ATTRIBUTES 00000000 12eca9d 000010 00 0 0 1

[30] .mdebug.abi32 PROGBITS 00000000 12ecaad 000000 00 0 0 1

[31] .shstrtab STRTAB 00000000 12ecaad 00013b 00 0 0 1

Key to Flags:

W (write), A (alloc), X (execute), M (merge), S (strings), I (info),

L (link order), O (extra OS processing required), G (group), T (TLS),

C (compressed), x (unknown), o (OS specific), E (exclude),

p (processor specific)

Enabling libunwind debug (Level=20) :

=======================================================

>unw_init_local_common: (cursor=0x775b7328)

>unw_init_local_common: (cursor=0x775b3328)

>access_mem: mem[55e737a0] -> 2002825

>get_rs_cache: acquiring lock

>put_rs_cache: unmasking signals/interrupts and releasing lock

>access_mem: mem[55869edc] -> 1040ffdf

>get_rs_cache: acquiring lock

>put_rs_cache: unmasking signals/interrupts and releasing lock

>access_mem: mem[5586a100] -> 10400009

>get_rs_cache: acquiring lock

>_ULmips_dwarf_find_proc_info: looking for IP=0x5586a0fb

>_ULmips_dwarf_callback: checking /opt/app/bin/myapp, base=0x555d3000)

>_ULmips_dwarf_callback: found table `/opt/app/bin/myapp': segbase=0x56761450, len=24246, gp=0x12e8940, table_data=0x5676145c

>lookup: e->start_ip_offset = ff6015d8

>lookup: e->start_ip_offset = ff327780

>lookup: e->start_ip_offset = ff1f9fc4

>lookup: e->start_ip_offset = ff1780f4

>lookup: e->start_ip_offset = ff148794

>lookup: e->start_ip_offset = ff12bcf8

>lookup: e->start_ip_offset = ff11ddc8

>lookup: e->start_ip_offset = ff113068

>lookup: e->start_ip_offset = ff10aa00

>lookup: e->start_ip_offset = ff108b38

>lookup: e->start_ip_offset = ff109a98

>lookup: e->start_ip_offset = ff1092c8

>lookup: e->start_ip_offset = ff108f70

>lookup: e->start_ip_offset = ff108bb0

>_ULmips_dwarf_search_unwind_table: ip=0x5586a0fb, start_ip=0xff108bb0

>_ULmips_dwarf_search_unwind_table: e->fde_offset = 27ebc, segbase = 56761450, debug_frame_base = 0, fde_addr = 5678930c

>_ULmips_dwarf_extract_proc_info_from_fde: FDE @ 0x5678930c

>_ULmips_dwarf_extract_proc_info_from_fde: looking for CIE at address 56789000

>parse_cie: CIE parsed OK, augmentation = "zR", handler=0x0

>_ULmips_dwarf_extract_proc_info_from_fde: FDE covers IP 0x5586a000-0x5586a0b0, LSDA=0x0

>put_rs_cache: unmasking signals/interrupts and releasing lock

[ALLOC] Alloc : 35 bytes : 2 : 0x55869ed8,0x5586a0fc,0,0,0,0,0

You should build your sources with

CFLAGS += -g1 -fno-omit-frame-pointer ...

LDFLAGS += -g1 -fno-omit-frame-pointer ... (if you are using LTO)

and then split executables (and libraries) to files with DWARF unwind tables

and files with debug symbols only; for each {executable}:

objcopy --only-keep-debug .../root/{executable} .../root/usr/lib/debug/{executable}.debug

objcopy --strip-debug .../root/{executable}

objcopy --add-gnu-debuglink=.../root/usr/lib/debug/{executable}.debug {executable}

where {executable} is a relative root path to a binary file (/usr/bin/exe for example).

You may not copy /usr/lib/debug to a target rootfs but use it to translate addresses to symbols on a host.

Check also that you built all linked libraries (including libc) with -g1 and -fno-omit-frame-pointer

to get full traces.

If you are using musl for MIPS, make sure you have a patch adding CFI directives to assembler sources.

For example https://github.com/keenetic/musl/commit/ee63340090e3261d04bc7e664f352a7e7dd1c32f

(based on http://lists.openwrt.org/pipermail/openwrt-devel/2020-June/029807.html).

On Mon, Jun 14, 2021 at 10:29 PM Story, Lenny <lstory@irobot.com> wrote:

Greetings,

The backtraces on my MIPS target are only going two levels, which happen to be declared inside my local file.

Because of its size, the target is stripped of the dwarf .debug_ * seconds, and the .symtab. Its just too big to load with symbols included. I am only looking to get the frame addresses from the target, and plan on performing the symbol resolution off target.

What should I look for in trying to see the trace back to main ?

Thanks !

-Lenny

This e-mail message and any files attached may contain information that iRobot Corporation considers confidential and/or proprietary or may later designate as confidential and proprietary. If you are not the intended recipient, please contact the sender and delete the email immediately. Unauthorized use or distribution is strictly prohibited.

From:	Story, Lenny
Subject:	Re: MIPS unwind only backtraces 2 local levels...
Date:	Wed, 16 Jun 2021 22:45:44 +0000