bug#54944: guix pull hangs in guix-packages-base.drv even with offloading

[Csepp]

Csepp <raingloom@riseup.net> writes:

> Csepp <raingloom@riseup.net> writes:
>
>> Csepp <raingloom@riseup.net> writes:
>>
>>> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>>>
>>>> Hi!
>>>>
>>>> raingloom <raingloom@riseup.net> writes:
>>>>
>>>>> It's been at 67% on guix-packages-base for at least an hour now. The
>>>>> system itself is responsive and with the swap I gave it, it has more
>>>>> than enough memory. Htop shows three guile processes at the top of the
>>>>> list when sorted by CPU%, their states are S, D, D.
>>>>> Both CPUs are practically idling.
>>>>> This looks like some kind of lockup to me.
>>>>>
>>>>> Fresh install based on bare-bones example on a 32 bit netbook, but the
>>>>> install image used is the latest tagged version, since apparently there
>>>>> is no 32 bit option for edge.
>>>>>
>>>>> I also tried pulling using channel-with-substitutes, since I'm not too
>>>>> keen on locally building everything on such an old machine. Although
>>>>> Guix itself should frankly not take this long to build if we want to be
>>>>> competitive with other distros. Anyways, pulling with that in
>>>>> channels.scm gives a cert related error, so that's great, means old
>>>>> images can't easily be used for installation.
>>>>
>>>> Have you been able to reproduce this?  If so, could you share the commit
>>>> you are starting from and the CPU architecture, so that we may hopefully
>>>> reproduce too?
>>>>
>>>> Thanks,
>>>>
>>>> Maxim
>>>
>>> CPU architecture is x86, commit it happened on last time is 347733b.
>>> Other possibly relevant factors:
>>> * spinning rust storage
>>> * 1GB RAM
>>> * encrypted BTRFS root
>>> * 4GB (encrypted) swap
>>> * 128MB zswap
>>>
>>> The last was not there when I originally submitted the bug.
>>>
>>> The swap is relevant because if it's a timing issue it's very possible
>>> some part of the code assumes reads are almost instant, which is not
>>> true with swap, and delaying a read might be exposing a race condition.
>>
>> Happening again.
>> pulled to: 8320c0c
>> pulled from: 4501a50
>>
>> Same system.
>>
>> The system version is from november of last year due, because trying to
>> upgrade takes so damn long and often gets stuck on some package with no
>> substitute.
>> So... the situation is not great...
>
> The process status says sleep so it's probably hanging in a syscall?
> Maybe a kernel bug?

Happening again with offloading.
This is getting really annoying.
Offload machine is completely idle, there is a process Guile for
guix-packages-base-builder running on it, its in sleeping status.  Ran
for 17 minutes, now the time is not increasing.
I'm attaching a GDB backtrace of all the threads.

Thread 9 (Thread 0xe4affac0 (LWP 878) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf794685e in __lll_lock_wait () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf794d11a in pthread_mutex_lock@@GLIBC_2.0 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7f1095c in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#4  0xf7f16f98 in scm_gensym () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#5  0xf5d3a8ef in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 8 (Thread 0xe5497ac0 (LWP 877) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf794685e in __lll_lock_wait () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf794d11a in pthread_mutex_lock@@GLIBC_2.0 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7f1095c in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#4  0xf7f16f98 in scm_gensym () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#5  0xf5d3a8ef in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 7 (Thread 0xe5c98ac0 (LWP 876) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf794685e in __lll_lock_wait () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf794d11a in pthread_mutex_lock@@GLIBC_2.0 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7f1095c in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#4  0xf7f16f98 in scm_gensym () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#5  0xf5d3a8ef in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xe6499ac0 (LWP 875) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf794685e in __lll_lock_wait () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf794d11a in pthread_mutex_lock@@GLIBC_2.0 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7f1095c in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#4  0xe32647e6 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xf5cf7ac0 (LWP 873) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf79c8237 in read () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf7e98292 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#3  0xf7e0a9c0 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#4  0xf7e0c14f in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#5  0xf7e1599c in GC_do_blocking () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#6  0xf7f11924 in scm_without_guile () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#7  0xf7ea0586 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#8  0xf7e8bf2d in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#9  0xf7ea2597 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#10 0xf7f1e1a1 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#11 0xf7f2e9ca in scm_call_n () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#12 0xf7e8d84f in scm_call_2 () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#13 0xf7f1cc55 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#14 0xf7f3ec20 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#15 0xf7f1876b in scm_c_catch () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#16 0xf7e8e63d in scm_c_with_continuation_barrier () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#17 0xf7f1779a in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#18 0xf7e15925 in GC_call_with_stack_base () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#19 0xf7f118ca in scm_with_guile () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#20 0xf7ea060f in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#21 0xf7949ee9 in start_thread () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#22 0xf79df458 in clone3 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6

Thread 4 (Thread 0xf6552ac0 (LWP 872) "GC-marker-2"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf79d26d2 in __libc_do_syscall () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf79464d9 in __futex_abstimed_wait_common () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7949474 in pthread_cond_wait@@GLIBC_2.3.2 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#4  0xf7e10056 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#5  0xf7e101b9 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#6  0xf7949ee9 in start_thread () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#7  0xf79df458 in clone3 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6

Thread 3 (Thread 0xf6d53ac0 (LWP 871) "GC-marker-1"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf79d26d2 in __libc_do_syscall () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf79464d9 in __futex_abstimed_wait_common () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7949474 in pthread_cond_wait@@GLIBC_2.3.2 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#4  0xf7e10056 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#5  0xf7e101b9 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#6  0xf7949ee9 in start_thread () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#7  0xf79df458 in clone3 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6

Thread 2 (Thread 0xf7554ac0 (LWP 870) "GC-marker-0"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf79d26d2 in __libc_do_syscall () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf79464d9 in __futex_abstimed_wait_common () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7949474 in pthread_cond_wait@@GLIBC_2.3.2 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#4  0xf7e10056 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#5  0xf7e101b9 in ?? () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#6  0xf7949ee9 in start_thread () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#7  0xf79df458 in clone3 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6

Thread 1 (Thread 0xf78c6700 (LWP 869) "guile"):
#0  0xf7fc6579 in __kernel_vsyscall ()
#1  0xf79d26d2 in __libc_do_syscall () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#2  0xf79464d9 in __futex_abstimed_wait_common () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#3  0xf7949474 in pthread_cond_wait@@GLIBC_2.3.2 () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#4  0xf7f11acc in scm_pthread_cond_wait () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#5  0xf7f15538 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#6  0xf7f17e9f in scm_timed_wait_condition_variable () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#7  0xf7ea25bd in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#8  0xf7f1e1a1 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#9  0xf7f2e9ca in scm_call_n () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#10 0xf7e8dbfa in scm_primitive_eval () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#11 0xf7ec7d1d in scm_primitive_load () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#12 0xf7ea2597 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#13 0xf7f1e1a1 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#14 0xf7f2e9ca in scm_call_n () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#15 0xf7e8dbfa in scm_primitive_eval () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#16 0xf7e93d32 in scm_eval () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#17 0xf7ef95a4 in scm_shell () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#18 0xf7ea541c in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#19 0xf7e8bf2d in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#20 0xf7ea2597 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#21 0xf7f1e1a1 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#22 0xf7f2e9ca in scm_call_n () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#23 0xf7e8d84f in scm_call_2 () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#24 0xf7f1cc55 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#25 0xf7f3ec20 in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#26 0xf7f1876b in scm_c_catch () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#27 0xf7e8e63d in scm_c_with_continuation_barrier () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#28 0xf7f1779a in ?? () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#29 0xf7e15925 in GC_call_with_stack_base () from 
target:/gnu/store/5zydbv97cly5mijfmnc6vrk2148qxd1n-libgc-8.2.2/lib/libgc.so.1
#30 0xf7f118ca in scm_with_guile () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#31 0xf7eaeb40 in scm_boot_guile () from 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/lib/libguile-3.0.so.1
#32 0x0804910c in ?? ()
#33 0xf78e9329 in __libc_start_call_main () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#34 0xf78e93ff in __libc_start_main_impl () from 
target:/gnu/store/0hr9jpczkcgpgqkhf4q4868xd57h5a62-glibc-2.35/lib/libc.so.6
#35 0x08049198 in ?? ()
Detaching from program: 
target:/gnu/store/da6ikq281d235hvb1cil2ls3iq80ni2m-guile-3.0.9/bin/guile, 
process 869
[Inferior 1 (process 869) detached]

System info:
offloading from: x86, 1 GB RAM, 4 GB swap, 2 cores, user guix commit is
8a47949, system commit is 038981e, commit being pulled is 01d5d68
offloading to:
amd64, 8 GB RAM, no swap, 4 cores
guix system commit is 9504dd2c3eef0277369acc0944f87fb4546251b1