libunwind-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Libunwind-devel] Remote unwinding support for perf on MIPS


From: Sapir Natan
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
Date: Mon, 30 Jul 2018 16:37:26 +0000

Hi Luke,

Thank you so much for your detailed reply.

We have the '.eh_frame' in our code.
We also located a problem when reaching the function 'perf_evsel__parse_sample' 
in file /tools/perf/util/evsel.c (linux 3.10)

We noticed that the field 'data->user_stack.size' is assign the value saved at 
'array += size / sizeof(*array)' which is 0 We changed the line to use the 
value stored in 'size' (which we checked to be the valid stack size), and now 
'perf' seems to create a valid callgraph This might be an issue in MIPS 
architecture, while all other supported architectures contain a valid argument 
at that offset (although I should note this is only a speculation at this point)

The fix on line 1176:
data->user_stack.size = *array; // Old line
data->user_stack.size = size;  // New line

Have you encountered any problems with this code section? 
Do you think it might be a register issue on MIPS?

Thanks,
Sapir

-----Original Message-----
From: Luke Diamand [mailto:address@hidden
Sent: Monday, July 30, 2018 11:57 AM
To: Sapir Natan
Cc: Sergey Korolev; Chen Muchtar; Roi Chen; address@hidden
Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

On 30 July 2018 at 07:10, Sapir Natan <address@hidden> wrote:
> Hi Luke,
>
>
>
> Thank you very much for your reply.
> I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’ 
> linked to perf or called from it at all (or anything from libunwind-ptrace).
>
> It is also already part of version 1.3-rc1, which means my perf still 
> fails to unwind even with this latest fix.
>

There are a ton of things that can go wrong along the way! I'm on vacation at 
the moment and I don't have easy access to the relevant source code, so this is 
all a bit from memory.

I used the 3 patches I think you mentioned to add perf MIPS support.

I've been getting to the point of capturing userspace callstacks (DWARF, not 
fp) using perf+libunwind on MIPSEL32, and it appears to work reasonably well, 
although I do see some callstacks possibly failing (most do, just a few 
mysteriously missing).

You need to make sure the code you are testing actually has any unwind tables 
in it - check the .eh_frame section is present.

I am using the current version of libunwind (1.3rc1 or thereabouts) and I don't 
think it will work with older versions. You need to make sure that you have the 
fix "dwarf: make dwarf_find_debug_frame public"
as otherwise perf just won't use libunwind at all.

I also found that in tdep-mips/libunwind_i.h, around line 150,
read_s32() seems to do some weird stuff with the address, which I think is for 
handling big-endian 64 bit addresses, but it doesn't seem to be the right thing 
to do on MIPSEL32. For now, I've just hacked out the calls to read_s32() and 
that seems to make it work - but you're actually on a 64 bit target so it might 
be different for you. I need to create a proper patch (sorry, not had time 
recently).

I also had no end of fun with split-debug (as generated by
OpenEmbedded) because the debug files end up in a place that libunwind does not 
expect to find them in. Watch out for that.

Another thing you will find is that with your version of perf, if you are 
running the unwinding on an x86-64 host, then most/all versions of perf will 
try to unwind the callstacks as though they are x86-64 callstacks rather than 
MIPS, which obviously doesn't work. From about linux 4.7 onwards there is 
cross-platform perf unwinding available, but the hooks for MIPS support have 
not been added (I don't yet have working hooks either). There are a couple of 
ways around this: you can run "perf report" on the target (but that is very 
slow) or you can run "perf report" on your x86-64 host under qemu-mips (you 
want the userland emulator not the full system version). This is what I've 
mostly been doing.

Make sure that libunwind has been built with --enable-debug-frame otherwise it 
won't work if your callstack goes through any code that was not compiled with 
-fasynchronous-unwind-tables or -fexceptions.

If you run "perf report -D" then it dumps out the file contents. Look through 
to see if you have any register dumps and check they are convincing looking 
MIPS register values.

Also look for the stack captures in the "perf report -D" report.
That's the giant 8kByte samples. If you haven't got any, then it's not doing 
the DWARF sampling at all.

Also check for the MMAP sections in the perf capture - sometimes perf can't 
read the /proc/PID/maps file of the thing it is trying to capture.

It sounds to me a bit as though you don't have a .eh_frame section in your 
image (there's a linker option as well that you need), and if perf/libunwind 
doesn't have access to the debug image, it won't work.
But it could also be quite a few other things!

Good luck!

Luke


>
>
> Did this patch fix the perf unwinding on MIPS issue for you?
>
> If so, can you tell me which patch you used to add this capability to 
> perf on MIPS? (patch to the linux code)
>
> Is there anything I’m missing here?
>
>
>
> Thanks,
>
> Sapir
>
>
>
> From: Luke Diamand [mailto:address@hidden
> Sent: Sunday, July 29, 2018 8:59 PM
> To: Sergey Korolev
> Cc: Sapir Natan; Chen Muchtar; Roi Chen; address@hidden
> Subject: Re: [Libunwind-devel] Remote unwinding support for perf on 
> MIPS
>
>
>
> I had a similar problem on mipsel32. At least on that the problem was 
> that the accessor function was calling read_s32 which seems to do the 
> wrong thing on that platform. Fixing that seems to have made it work.
> I'm in the process of trying to organise a patch.
>
>
>
> On Sun, 29 Jul 2018, 18:25 Sergey Korolev, <address@hidden> wrote:
>
> In my opinion a mentioned patch [1] does not work properly
>
> since it defines _UPT_reg_offset elements as
>
>
> +    [UNW_MIPS_R0]  =  0,
> +    [UNW_MIPS_R1]  =  1,
> +    [UNW_MIPS_R2]  =  2,
> +    [UNW_MIPS_R3]  =  3,
> +    [UNW_MIPS_R4]  =  4,
> +    [UNW_MIPS_R5]  =  5,
> +    [UNW_MIPS_R6]  =  6,
> +    [UNW_MIPS_R7]  =  7,
> +    [UNW_MIPS_R8]  =  8,
> +    [UNW_MIPS_R9]  =  9,
> +    [UNW_MIPS_R10] = 10,
> +    [UNW_MIPS_R11] = 11,
> +    [UNW_MIPS_R12] = 12,
> +    [UNW_MIPS_R13] = 13,
> +    [UNW_MIPS_R14] = 14,
> +    [UNW_MIPS_R15] = 15,
> +    [UNW_MIPS_R16] = 16,
> +    [UNW_MIPS_R17] = 17,
> +    [UNW_MIPS_R18] = 18,
> +    [UNW_MIPS_R19] = 19,
> +    [UNW_MIPS_R20] = 20,
> +    [UNW_MIPS_R21] = 21,
> +    [UNW_MIPS_R22] = 22,
> +    [UNW_MIPS_R23] = 23,
> +    [UNW_MIPS_R24] = 24,
> +    [UNW_MIPS_R25] = 25,
> +    [UNW_MIPS_R26] = 26,
> +    [UNW_MIPS_R27] = 27,
> +    [UNW_MIPS_R28] = 28,
> +    [UNW_MIPS_R29] = 29,
> +    [UNW_MIPS_R30] = 30,
> +    [UNW_MIPS_R31] = 31,
> +    [UNW_MIPS_PC]  = 64,
>
>
>
> but a code in _UPT_access_mem.c expects that _UPT_reg_offset
>
> is a byte offset, and not a word offset or a register index.
>
>
>
> [1]
> http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a6
> 29634b3225fccb1ed8a0a765bccc77f2e
>
>
>
> On Sun, Jul 29, 2018 at 6:18 PM, Sapir Natan <address@hidden> wrote:
>
> Hi,
>
>
>
> We are trying to obtain a callgraph using ‘perf’ utility, while 
> running on MIPS architecture with linux 3.10 (64Bit processes)
>
> We wish to obtain callgraph on:
>
> ·         Terminated processes
>
> ·         Running processes (using –p <pid>)
>
>
>
> Following the resolution that perf is unable to create callgraph on 
> MIPS using frame pointer, we tried using libunwind
>
> Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other 
> version), we added a patch (composed of the following 3 patches):
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
>
> ·         As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
> source version 1.2.1).
>
>
>
> This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we 
> still couldn’t obtain a usable callgraph (most functions had no trace)
>
> The problem we encountered was that when reaching ‘maps__find´ 
> function (linux/tools/perf/util/map.c), the binary search for the 
> required entry failed (could not be located)
>
> From our investigation, we can support local unwinding but not remote 
> (which is the type of unwinding required by ‘perf’)
>
>
>
> Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
> for version 1.3.
>
> We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however 
> the results remained the same (no callgraph).
>
>
>
> Could you please assist?
>
> ·         What was the issue you encountered that the fix in 1.3 version
> resolved?
>
> ·         Can the new addition enable ‘perf’ to support callgraph on MIPS?
>
> ·         Did we miss anything in our steps (as described above)?
>
> ·         Can you advice on the next steps?
>
>
>
> Thanks,
>
> Sapir
>
>
>
>
> _______________________________________________
> Libunwind-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>
>
>
> _______________________________________________
> Libunwind-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>
>
>
> Email secured by Check Point.

Email secured by Check Point.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]