Re: Performance of computing cross derivations

[Christopher Baines]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> When asked by the data service, it seems to take Guix around 3 minutes
>> to compute cross derivations for all packages (to a single
>> target). Here's a simple script that replicates this:

...

> One idiom that defeats caching is:
>
>   (define (make-me-a-package x y z)
>     (package
>       …))
>
> Such a procedure returns a fresh package every time it’s called,
> preventing caching from happening (because cache entries are compared
> with ‘eq?’).  That typically leads to lower hit rates.
>
> Anyway, lots of words to say that I don’t see anything immediately
> obvious with cross-compilation, yet I wouldn’t be surprised if some of
> these cache-defeating idioms were used because we’ve payed less
> attention to this.

I've got a feeling that performance has got worse since I looked at this
originally, I've finally got around to having a further look.

I spent some time looking at various metrics, but it was most useful to
just write the cache keys of various types to files and have a read.

The cross-base module was causing many issues, as all but one of the
procedures there produced new package records each time. There is also
make-rust-sysroot which showed up.

I've sent some patches as #68266 to add memoization to avoid this, and
that seems to speed things up.

Looking at other things in the cache, I think there are some issues with
file-append and local-file. The use of file-append in svn-fetch and
local-file in the lower procedure in the python build system both bloat
the cache for example, although I'm less sure about how to address these
cases.

One thing I am sure about though, is that these problems will come
back. Maybe we could add some reporting in to Guix to look through the
cache at the keys, lower them all and check for equivalence. That way it
should be possible to automate saying that having [1] in the cache
several thousand times is unhelpful. The data service could then run
this reporting and store it.

1: #<file-append #<package subversion@1.14.2 
gnu/packages/version-control.scm:2267 7f294d908840> "/bin/svn">

signature.asc
Description: PGP signature