Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

[Luke Diamand]

On 30 July 2018 at 07:10, Sapir Natan <address@hidden> wrote:
> Hi Luke,
>
>
>
> Thank you very much for your reply.
> I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’ linked
> to perf or called from it at all (or anything from libunwind-ptrace).
>
> It is also already part of version 1.3-rc1, which means my perf still fails
> to unwind even with this latest fix.
>

There are a ton of things that can go wrong along the way! I'm on
vacation at the moment and I don't have easy access to the relevant
source code, so this is all a bit from memory.

I used the 3 patches I think you mentioned to add perf MIPS support.

I've been getting to the point of capturing userspace callstacks
(DWARF, not fp) using perf+libunwind on MIPSEL32, and it appears to
work reasonably well, although I do see some callstacks possibly
failing (most do, just a few mysteriously missing).

You need to make sure the code you are testing actually has any unwind
tables in it - check the .eh_frame section is present.

I am using the current version of libunwind (1.3rc1 or thereabouts)
and I don't think it will work with older versions. You need to make
sure that you have the fix "dwarf: make dwarf_find_debug_frame public"
as otherwise perf just won't use libunwind at all.

I also found that in tdep-mips/libunwind_i.h, around line 150,
read_s32() seems to do some weird stuff with the address, which I
think is for handling big-endian 64 bit addresses, but it doesn't seem
to be the right thing to do on MIPSEL32. For now, I've just hacked out
the calls to read_s32() and that seems to make it work - but you're
actually on a 64 bit target so it might be different for you. I need
to create a proper patch (sorry, not had time recently).

I also had no end of fun with split-debug (as generated by
OpenEmbedded) because the debug files end up in a place that libunwind
does not expect to find them in. Watch out for that.

Another thing you will find is that with your version of perf, if you
are running the unwinding on an x86-64 host, then most/all versions of
perf will try to unwind the callstacks as though they are x86-64
callstacks rather than MIPS, which obviously doesn't work. From about
linux 4.7 onwards there is cross-platform perf unwinding available,
but the hooks for MIPS support have not been added (I don't yet have
working hooks either). There are a couple of ways around this: you can
run "perf report" on the target (but that is very slow) or you can run
"perf report" on your x86-64 host under qemu-mips (you want the
userland emulator not the full system version). This is what I've
mostly been doing.

Make sure that libunwind has been built with --enable-debug-frame
otherwise it won't work if your callstack goes through any code that
was not compiled with -fasynchronous-unwind-tables or -fexceptions.

If you run "perf report -D" then it dumps out the file contents. Look
through to see if you have any register dumps and check they are
convincing looking MIPS register values.

Also look for the stack captures in the "perf report -D" report.
That's the giant 8kByte samples. If you haven't got any, then it's not
doing the DWARF sampling at all.

Also check for the MMAP sections in the perf capture - sometimes perf
can't read the /proc/PID/maps file of the thing it is trying to
capture.

It sounds to me a bit as though you don't have a .eh_frame section in
your image (there's a linker option as well that you need), and if
perf/libunwind doesn't have access to the debug image, it won't work.
But it could also be quite a few other things!

Good luck!

Luke


>
>
> Did this patch fix the perf unwinding on MIPS issue for you?
>
> If so, can you tell me which patch you used to add this capability to perf
> on MIPS? (patch to the linux code)
>
> Is there anything I’m missing here?
>
>
>
> Thanks,
>
> Sapir
>
>
>
> From: Luke Diamand [mailto:address@hidden
> Sent: Sunday, July 29, 2018 8:59 PM
> To: Sergey Korolev
> Cc: Sapir Natan; Chen Muchtar; Roi Chen; address@hidden
> Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
>
>
>
> I had a similar problem on mipsel32. At least on that the problem was that
> the accessor function was calling read_s32 which seems to do the wrong thing
> on that platform. Fixing that seems to have made it work. I'm in the process
> of trying to organise a patch.
>
>
>
> On Sun, 29 Jul 2018, 18:25 Sergey Korolev, <address@hidden> wrote:
>
> In my opinion a mentioned patch [1] does not work properly
>
> since it defines _UPT_reg_offset elements as
>
>
> +    [UNW_MIPS_R0]  =  0,
> +    [UNW_MIPS_R1]  =  1,
> +    [UNW_MIPS_R2]  =  2,
> +    [UNW_MIPS_R3]  =  3,
> +    [UNW_MIPS_R4]  =  4,
> +    [UNW_MIPS_R5]  =  5,
> +    [UNW_MIPS_R6]  =  6,
> +    [UNW_MIPS_R7]  =  7,
> +    [UNW_MIPS_R8]  =  8,
> +    [UNW_MIPS_R9]  =  9,
> +    [UNW_MIPS_R10] = 10,
> +    [UNW_MIPS_R11] = 11,
> +    [UNW_MIPS_R12] = 12,
> +    [UNW_MIPS_R13] = 13,
> +    [UNW_MIPS_R14] = 14,
> +    [UNW_MIPS_R15] = 15,
> +    [UNW_MIPS_R16] = 16,
> +    [UNW_MIPS_R17] = 17,
> +    [UNW_MIPS_R18] = 18,
> +    [UNW_MIPS_R19] = 19,
> +    [UNW_MIPS_R20] = 20,
> +    [UNW_MIPS_R21] = 21,
> +    [UNW_MIPS_R22] = 22,
> +    [UNW_MIPS_R23] = 23,
> +    [UNW_MIPS_R24] = 24,
> +    [UNW_MIPS_R25] = 25,
> +    [UNW_MIPS_R26] = 26,
> +    [UNW_MIPS_R27] = 27,
> +    [UNW_MIPS_R28] = 28,
> +    [UNW_MIPS_R29] = 29,
> +    [UNW_MIPS_R30] = 30,
> +    [UNW_MIPS_R31] = 31,
> +    [UNW_MIPS_PC]  = 64,
>
>
>
> but a code in _UPT_access_mem.c expects that _UPT_reg_offset
>
> is a byte offset, and not a word offset or a register index.
>
>
>
> [1]
> http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a629634b3225fccb1ed8a0a765bccc77f2e
>
>
>
> On Sun, Jul 29, 2018 at 6:18 PM, Sapir Natan <address@hidden> wrote:
>
> Hi,
>
>
>
> We are trying to obtain a callgraph using ‘perf’ utility, while running on
> MIPS architecture with linux 3.10 (64Bit processes)
>
> We wish to obtain callgraph on:
>
> ·         Terminated processes
>
> ·         Running processes (using –p <pid>)
>
>
>
> Following the resolution that perf is unable to create callgraph on MIPS
> using frame pointer, we tried using libunwind
>
> Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
> version), we added a patch (composed of the following 3 patches):
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
>
> ·
> https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
>
> ·         As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
> source version 1.2.1).
>
>
>
> This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we still
> couldn’t obtain a usable callgraph (most functions had no trace)
>
> The problem we encountered was that when reaching ‘maps__find´ function
> (linux/tools/perf/util/map.c), the binary search for the required entry
> failed (could not be located)
>
> From our investigation, we can support local unwinding but not remote (which
> is the type of unwinding required by ‘perf’)
>
>
>
> Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
> for version 1.3.
>
> We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however the
> results remained the same (no callgraph).
>
>
>
> Could you please assist?
>
> ·         What was the issue you encountered that the fix in 1.3 version
> resolved?
>
> ·         Can the new addition enable ‘perf’ to support callgraph on MIPS?
>
> ·         Did we miss anything in our steps (as described above)?
>
> ·         Can you advice on the next steps?
>
>
>
> Thanks,
>
> Sapir
>
>
>
>
> _______________________________________________
> Libunwind-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>
>
>
> _______________________________________________
> Libunwind-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>
>
>
> Email secured by Check Point.