Re: [PATCH RFC] qemu co-mutex crash / question

[Vladimir Sementsov-Ogievskiy]

09.12.2020 15:32, Vladimir Sementsov-Ogievskiy wrote:
> test-aio-multithread: ../util/qemu-coroutine-lock.c:197: qemu_co_mutex_wake: 
> Assertion `mutex == co->wait_on_mutex' failed.
>
>      Thread 18 "test-aio-multit" received signal SIGABRT, Aborted.
>      [Switching to Thread 0x7fffe5ffb700 (LWP 24549)]
>      0x00007ffff7063625 in raise () from /lib64/libc.so.6
>      (gdb) bt
>      #0  0x00007ffff7063625 in raise () from /lib64/libc.so.6
>      #1  0x00007ffff704c8d9 in abort () from /lib64/libc.so.6
>      #2  0x00007ffff704c7a9 in __assert_fail_base.cold () from /lib64/libc.so.6
>      #3  0x00007ffff705ba66 in __assert_fail () from /lib64/libc.so.6
>      #4  0x000055555568c153 in qemu_co_mutex_wake (mutex=0x555555771360 
> <comutex>, co=0x555555803ec0) at ../util/qemu-coroutine-lock.c:197
>      #5  0x000055555568c5a0 in qemu_co_mutex_unlock (mutex=0x555555771360 
> <comutex>) at ../util/qemu-coroutine-lock.c:307
>      #6  0x000055555557acfd in test_multi_co_mutex_entry (opaque=0x0) at 
> ../tests/test-aio-multithread.c:208
>      #7  0x00005555556bb5d7 in coroutine_trampoline (i0=1434467712, i1=21845) 
> at ../util/coroutine-ucontext.c:173
>      #8  0x00007ffff7078d30 in ?? () from /lib64/libc.so.6
>      #9  0x00007fffffffd850 in ?? ()
>      #10 0x0000000000000000 in ?? ()
>      (gdb) fr 4
>      #4  0x000055555568c153 in qemu_co_mutex_wake (mutex=0x555555771360 
> <comutex>, co=0x555555803ec0) at ../util/qemu-coroutine-lock.c:197
>      197         assert(mutex == co->wait_on_mutex);
>      (gdb) p mutex
>      $1 = (CoMutex *) 0x555555771360 <comutex>
>      (gdb) p co->wait_on_mutex
>      $2 = (CoMutex *) 0x555555771360 <comutex>
>      (gdb) p mutex == co->wait_on_mutex
>      $3 = 1
>
> So, it failed, but in gdb the condition is true.. How can that be?

Interesting: I tried to run test on one cpu:

for i in {1..100}; do taskset -c 0 ./build/tests/test-aio-multithread -p 
/aio/multi/mutex/handoff; done

with taskset it takes a lot more tries to reproduce, but finally I have correct 
coredump with correct assertion failure:

(gdb) bt
#0  0x00007ff7fa22d625 in raise () from /lib64/libc.so.6
#1  0x00007ff7fa2168d9 in abort () from /lib64/libc.so.6
#2  0x00007ff7fa2167a9 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ff7fa225a66 in __assert_fail () from /lib64/libc.so.6
#4  0x0000564c7ca99153 in qemu_co_mutex_wake (mutex=0x564c7cb7e360 <comutex>, 
co=0x564c7d3f5c40) at ../util/qemu-coroutine-lock.c:197
#5  0x0000564c7ca995a0 in qemu_co_mutex_unlock (mutex=0x564c7cb7e360 <comutex>) 
at ../util/qemu-coroutine-lock.c:307
#6  0x0000564c7c987cfd in test_multi_co_mutex_entry (opaque=0x0) at 
../tests/test-aio-multithread.c:208
#7  0x0000564c7cac85d7 in coroutine_trampoline (i0=2101304064, i1=22092) at 
../util/coroutine-ucontext.c:173
#8  0x00007ff7fa242d30 in ?? () from /lib64/libc.so.6
#9  0x00007ffd3b3c6ac0 in ?? ()
#10 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7ff7ed19c000
(gdb) fr 4
#4  0x0000564c7ca99153 in qemu_co_mutex_wake (mutex=0x564c7cb7e360 <comutex>, 
co=0x564c7d3f5c40) at ../util/qemu-coroutine-lock.c:197
197         assert(mutex == co->wait_on_mutex);
(gdb) p mutex
$1 = (CoMutex *) 0x564c7cb7e360 <comutex>
(gdb) p co->wait_on_mutex
$2 = (CoMutex *) 0x0


other interesting threads:

Thread 7 (Thread 0x7ff7ef19f700 (LWP 261134)):
#0  0x0000564c7ca98f99 in push_waiter (mutex=0x564c7cb7e360 <comutex>, 
w=0x7ff7ed09aea0) at ../util/qemu-coroutine-lock.c:151
#1  0x0000564c7ca991c4 in qemu_co_mutex_lock_slowpath (ctx=0x7ff7e4000b60, 
mutex=0x564c7cb7e360 <comutex>) at ../util/qemu-coroutine-lock.c:211
#2  0x0000564c7ca993f5 in qemu_co_mutex_lock (mutex=0x564c7cb7e360 <comutex>) 
at ../util/qemu-coroutine-lock.c:277
#3  0x0000564c7c987ce2 in test_multi_co_mutex_entry (opaque=0x0) at 
../tests/test-aio-multithread.c:206
#4  0x0000564c7cac85d7 in coroutine_trampoline (i0=2101304384, i1=22092) at 
../util/coroutine-ucontext.c:173
#5  0x00007ff7fa242d30 in ?? () from /lib64/libc.so.6
#6  0x00007ffd3b3c6ac0 in ?? ()
#7  0x0000000000000000 in ?? ()

#0  0x00007ff7fa3cdf55 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007ff7fb0d27b7 in g_usleep () from /lib64/libglib-2.0.so.0
#2  0x0000564c7c987e05 in test_multi_co_mutex (threads=2, seconds=3) at 
../tests/test-aio-multithread.c:237
#3  0x0000564c7c987eff in test_multi_co_mutex_2_3 () at 
../tests/test-aio-multithread.c:270
#4  0x00007ff7fb0cface in g_test_run_suite_internal () from 
/lib64/libglib-2.0.so.0
#5  0x00007ff7fb0cf874 in g_test_run_suite_internal () from 
/lib64/libglib-2.0.so.0
#6  0x00007ff7fb0cf874 in g_test_run_suite_internal () from 
/lib64/libglib-2.0.so.0
#7  0x00007ff7fb0cf874 in g_test_run_suite_internal () from 
/lib64/libglib-2.0.so.0
#8  0x00007ff7fb0cff7b in g_test_run_suite () from /lib64/libglib-2.0.so.0
#9  0x00007ff7fb0cffd5 in g_test_run () from /lib64/libglib-2.0.so.0
#10 0x0000564c7c98874e in main (argc=1, argv=0x7ffd3b3c7868) at 
../tests/test-aio-multithread.c:459



--
Best regards,
Vladimir