[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#49439: grafts cause “guix environment” to get killed with OOM
From: |
Sarah Morgensen |
Subject: |
bug#49439: grafts cause “guix environment” to get killed with OOM |
Date: |
Wed, 28 Jul 2021 20:20:36 -0700 |
Hi,
Ludovic Courtès <ludo@gnu.org> writes:
> Hi,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Thinking about it, the grafts code depends on what’s in the store: when
>> nothing is in the store, it bounces to the “build handler”, which
>> accumulates the list of missing store items, until it starts building
>> them.
>
> So I can reproduce the problem Ricardo and you initially reported by
> running:
>
> ./pre-inst-env guix environment pigx-scrnaseq --search-paths
>
> after removing some of the ungrafted store items with:
>
> guix gc -D $(guix build r-rlang --no-grafts) \
> $(guix gc --references $(guix build pigx-scrnaseq --no-grafts))
Same here. I'm glad we were able to pinpoint this!
>
> The seemingly endless CPU usage and unbound memory use comes from the
> ‘build-accumulator’ build handler. Here, the graph of ‘pigx-scrnaseq’
> has many nodes, and many of them depend on, say, ‘r-rlang’. Thus, when
> ‘r-rlang’ is not in the store, the grafting code keeps asking for it by
> calling ‘build-derivations’, which aborts to the build handler; the
> build handler saves the .drv file name and the continuation and keeps
> going. But since the next package also depends on ‘r-langr’, we abort
> again to the build handler, and so on.
>
> The end result is a very long list of <unresolved> nodes, probably of
> this order in this case:
>
> $ guix graph -t reverse-package r-rlang |grep 'label = "'|wc -l
> 594
>
> Presumably, the captured continuations occupy quite a bit of memory,
> hence the quick growth.
>
> I suppose one solution is to fire suspended builds when the build
> handler sees a build request for a given derivation for the second time.
> It needs more thought and testing…
I wonder if instead it's possible to reduce the memory taken by the
continuations? As someone who has absolutely no experience with the
build/derivation system, it seems like all we *should* need to save is
the derivation file name.
>
> Ludo’.
>
> PS: Did you know ‘pigx-scrnaseq’ has twice as many nodes as
> ‘libreoffice’?
>
> $ guix graph -t bag pigx-scrnaseq |grep 'label = "'|wc -l
> 1359
> $ guix graph -t bag libreoffice |grep 'label = "'|wc -l
> 699
>
> That makes it a great example to study and fix scalability issues!
For those interested, I've compiled a list of the top 60
highest-dependency packages, as reported by package-closure:
pigx 1630
nextcloud-client 1539
akregator 1486
kmail 1484
korganizer 1481
kdepim-runtime 1480
kincidenceeditor 1478
keventviews 1477
kmailcommon 1476
kcalendarsupport 1475
kmessagelib 1474
knotes 1472
kaddressbook 1469
libksieve 1467
kdepim-apps-libs 1465
libgravatar 1463
kpimcommon 1462
akonadi-calendar 1453
pigx-bsseq 1448
elisa 1446
kaffeine 1432
kdenlive 1431
kmailtransport 1431
dolphin-plugins 1426
k3b 1424
libkgapi 1422
dolphin 1421
kopete 1403
pigx-sars-cov2-ww 1401
krdc 1398
baloo-widgets 1397
baloo 1396
pigx-chipseq 1396
krfb 1389
ffmpegthumbs 1388
kget 1382
kmplayer 1381
kdelibs4support 1375
pigx-scrnaseq 1342
kdevelop 1340
kmailimporter 1326
libkdepim 1325
pigx-rnaseq 1324
labplot 1316
smb4k 1315
kleopatra 1311
kalarmcal 1311
choqok 1311
okular 1310
ktnef 1310
ktorrent 1310
kate 1308
akonadi-search 1308
kcalutils 1307
yakuake 1306
khelpcenter 1305
libksysguard 1305
kdeconnect 1304
konsole 1304
libkleo 1304
There seem to be a lot of kde packages in there, so perhaps this issue
isn't as rare as we might otherwise expect?
--
Sarah