[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#65033: [PATCH] guix: read-derivation-from-file: Use less open files.
From: |
Christopher Baines |
Subject: |
bug#65033: [PATCH] guix: read-derivation-from-file: Use less open files. |
Date: |
Tue, 12 Sep 2023 09:29:23 +0100 |
User-agent: |
mu4e 1.10.5; emacs 28.2 |
Ludovic Courtès <ludo@gnu.org> writes:
> Hey Chris,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> The Guix derivation graph isn't that deep, so I don't think this generally
>> opens lots of files, but I think it's still unnecessary to keep more files
>> than needed open.
>>
>> * guix/derivations.scm (read-derivation-from-file): Read each derivation to a
>> string, which is passed as a port to read-derivation.
>
> [...]
>
>> - (let ((drv (call-with-input-file file read-derivation)))
>> + (let ((drv
>> + ;; read-derivation can call read-derivation-from-file, so to
>> + ;; avoid having multiple open files when reading a derivation
>> + ;; with inputs, read it in to a string first.
>> + (call-with-input-string
>> + (call-with-input-file file
>> + get-string-all)
>> + read-derivation)))
>
> How real is the risk of having too many open files due to a
> ‘read-derivation’ call? (Where too many is >= 100.)
>
> You might think it’s likely because:
>
> $ guix gc -R $(guix build -d --no-grafts emacs) |grep drv$ | wc -l
> 2234
>
>
> But in fact, due to the shape of the graph + memoization (I suppose¹),
> there are at most 27 open file descriptors in this case:
>
> $ strace -o /tmp/log.strace -e openat guile -c "(use-modules (guix))
> (read-derivation-from-file \"$(guix build -d --no-grafts emacs)\")"
> $ cut -d '=' -f 2- < /tmp/log.strace |sort -un | tail
> 18
> 19
> 20
> 21
> 22
> 23
> 24
> 25
> 26
> 27
>
>
> So to me the status quo is probably okay.
>
> The reason I’m paying attention to this is that allocating a string port
> plus a string for the whole contents every time would put pressure on
> memory, which is worth avoiding if we can.
>
> WDYT?
I guess there should be a way of arranging the code so that it doesn't
keep unnecessary ports, but also doesn't use strings, but that will
require some rearranging.
I think I just got thinking about this as the build coordinator was
using excessive file descriptors, but this isn't the cause.
signature.asc
Description: PGP signature
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- bug#65033: [PATCH] guix: read-derivation-from-file: Use less open files.,
Christopher Baines <=