[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran
|
From: |
Ludovic Courtès |
|
Subject: |
bug#63852: ‘guix-package-cache.drv’ eats all your memory with ‘guix-cran’ |
|
Date: |
Sat, 03 Jun 2023 16:05:08 +0200 |
|
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) |
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
> It goes OK until building
> /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-guix-package-cache.drv, which
> consumes memory until it gets OOM-killed.
The culprit is easily found (here on just the 22K packages of ‘guix’, so
less than half of what you get with ‘guix-cran’):
--8<---------------cut here---------------start------------->8---
$ guix repl
GNU Guile 3.0.9
Copyright (C) 1995-2023 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> ,m(gnu packages)
scheme@(gnu packages)> ,use(statprof)
scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")))
% cumulative self
time seconds seconds procedure
50.00 13.77 13.77 ice-9/boot-9.scm:2498:4
6.94 1.91 1.91 display
5.56 1.53 1.53 object->string
2.78 0.76 0.76 reverse
2.78 0.76 0.76 gensym
2.78 0.76 0.76 ice-9/boot-9.scm:2217:0:%load-announce
1.39 6.88 0.38 gnu/packages.scm:388:4:expand-cache
1.39 0.76 0.38 srfi/srfi-1.scm:1028:0:lset-intersection
1.39 0.38 0.38 number->string
1.39 0.38 0.38 gnu/packages/crates-io.scm:17783:2:arguments
1.39 0.38 0.38 open-output-string
1.39 0.38 0.38 gnu/packages/crates-io.scm:18450:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:36208:2:arguments
1.39 0.38 0.38 guix/packages.scm:1320:8:mproc
1.39 0.38 0.38 gnu/packages/crates-io.scm:25101:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:29972:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:883:2:arguments
1.39 0.38 0.38 ice-9/vlist.scm:449:0:vhash-cons
1.39 0.38 0.38 append
1.39 0.38 0.38 string-append
1.39 0.38 0.38 system/vm/assembler.scm:2175:8
1.39 0.38 0.38
guix/build/syscalls.scm:1168:0:read-dirent-header/linux
1.39 0.38 0.38 srfi/srfi-1.scm:1033:17
1.39 0.38 0.38 gnu/packages/crates-io.scm:9336:2:arguments
1.39 0.38 0.38 gnu/packages/crates-io.scm:64009:2:arguments
1.39 0.38 0.38 guix/build-system/cargo.scm:229:4
1.39 0.38 0.38 ice-9/boot-9.scm:2759:0:module-make-local-var!
0.00 7328.64 0.00 system/vm/assembler.scm:1258:0:intern-constant
0.00 1423.97 0.00 ice-9/boot-9.scm:220:5:map1
0.00 201.90 0.00 ice-9/threads.scm:388:4
[...]
---
Sample count: 72
Total time: 27.531145665 seconds (14.056589389 seconds in GC)
--8<---------------cut here---------------end--------------->8---
The assembler allocates a huge constant table in ‘intern-constant’
(‘generate-package-cache’ produces one literal list containing one small
vector per package, each of which contains strings and other constants).
The attached file emulates what ‘generate-package-cache’ does. Each
vector in the list looks like this:
#(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f
"gnu/packages/xyz0" 0 1)
The program terminates with roughly a 250 MiB heap for 22K entries—about
10 MiB per vector.
The heap profile on completion looks like this:
--8<---------------cut here---------------start------------->8---
% type self avg obj size
57.5 bytevector 8,205,632 455868.4
23.9 vector 3,413,857 549.3
7.7 struct 1,093,840 40.8
5.3 pair 753,376 16.0
2.7 stringbuf 385,856 51.1
1.1 symbol 151,424 32.0
0.4 program 59,424 47.3
0.3 string 49,152 32.0
0.2 heap-number 23,520 32.0
0.2 variable 21,696 30.3
0.1 smob 21,056 32.0
0.1 vm-continuation 20,864 32.0
0.1 atomic-box 20,736 32.0
0.1 weak-table 19,584 31.8
0.1 unknown 19,040 32.0
0.0 pointer 2,000 16.0
0.0 hash-table 896 32.0
0.0 primitive 144 16.0
0.0 weak-vector 96 16.0
0.0 frame 96 32.0
0.0 primitive-generic 64 32.0
0.0 keyword 16 16.0
sampled heap: 13.60166 MiB (heap size: 233.16016 MiB)
--8<---------------cut here---------------end--------------->8---
It’s likely that the ‘buf’ field of <asm> is what’s taking space
On that topic, see also:
https://lists.gnu.org/archive/html/guile-devel/2023-01/msg00013.html
Ludo’.
;; https://issues.guix.gnu.org/63852
(use-modules (system base compile)
(language tree-il)
(srfi srfi-1)
(srfi srfi-26)
(ice-9 match)
(ice-9 time)
(statprof))
(define lst
(unfold (cut > <> 22000)
(lambda (i)
;; Mimic 'generate-package-cache'.
(define name
(string-append "xyzxyz-" (number->string i 8)))
(vector i name
(string-append "1.0."
(number->string (modulo i 300)))
(list 'gnu 'packages
(string->symbol
(string-append "xyz" (number->string i))))
(string->symbol name)
'("out")
#t
#f
(string-append "gnu/packages/xyz"
(number->string (modulo i 300)))
(modulo i 1000)
1))
1+
0))
(pk 'go! (gc-stats))
(let ((exp (match (or (getenv "VARIANT") "0")
("0" `',lst)
("1" `(list ,@lst))
("2" (make-const #f lst)) ;avoid psyntax overhead
("3" (make-primcall #f 'list
(map (lambda (vector)
(make-const #f vector))
lst))))))
(compile exp #:to 'bytecode
#:from (if (pair? exp)
'scheme
'tree-il)
#:optimization-level 0
#:opts '(#:to-file? #t)))
(pk (gc-stats))