gluster-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Gluster-devel] debugging syntasks


From: Anand Avati
Subject: [Gluster-devel] debugging syntasks
Date: Sat, 4 May 2013 00:30:34 -0700

Summarizing some steps which might help others who are debugging synctask related issues. Since synctasks uses swapcontext() to switch among themselves, the pthread stack would only be representing the current executing tasks stack alone (which usually would be the syncenv_processor() scheduler.) To inspect the "stack backtrace" of other synctasks:

0. Attach gdb to the live running (possibly deadlocked or hung) process. This approach does not work on a core dump (there might be a way to apply the same principles on a core dump, any suggestions welcome)

1. Get the address of their 'struct synctask'. All the yielded synctasks are linked in a list originating at syncenv->waitq. The addresses of the links can be typecasted into 'struct synctask *' (as long as we are keeping the first member as 'all_tasks' in the structure definition).

2. Once you have the address of the 'struct synctask' you want to inspect the backtrace, inspect basic sanity of the structure with:

(gdb) p *(struct synctask *) 0xaddre55

Verify that the structure looks reasonably alright.

3. Perform a quick "sneak peek" into that synctask's backtrace at the time it yielded away:

(gdb) set $rsp = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[15]
(gdb) set $pc = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[16]
(gdb) bt

Note that you are essentially "sacrificing" the current thread in order to inspect the backtrace. So to be safe, you would want to

a) save the original $rsp and $pc values
b) NOT execute "continue" after setting the synctask's saved values. Instead just inspect the stack with commands like "bt", "frame" etc. and revert back the orig $pc and $rsp variables.

4. The above step is for x86_64. For other architectures you will need to find out the appropriate names of registers in gdb for the stack pointer and program counter, and also the index number of those saved register values inside gregs[].

For i386:

(gdb) set $esp = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[7]
(gdb) set $pc = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[14]

For your architecture, you will need to confirm by inspecting /usr/include/sys/ucontext.h for the actual offsets of the saved register values.

5. Note that this procedure is not really a "complete context switch". For a full context switch you will need to load all the saved register values necessary. This is because the compiler may
chose to maintain variables completely in the registers and not on the stack - more so common in x86_64 than i386. Compiling with -O0 should "improve the situation" most of the times, but not a guarantee. You can load the other registers from the saved values too (look up the appropriate index into gregs[] for the value from /usr/include/sys/ucontext.h), but remember to save the current register values if you care about restoring and resuming the stalled process after inspection.

Happy hacking!
Avati


reply via email to

[Prev in Thread] Current Thread [Next in Thread]