Re: Guix pull speed

[Ludovic Courtès]

Hello!

Josselin Poiret <dev@jpoiret.xyz> skribis:

> After looking a bit more into guix pull speed, or to be more precise the
> "Computing Guix derivation..." step, which is not substitutable.  I've
> come to the conclusion that the thing that takes the majority of the
> time is loading the files that define the packages that the new Guix
> needs to build itself.  These files are not compiled yet, and worse,
> loading just (gnu packages guile) ends up loading 361 other package
> files.  You can generate a package graph in GraphML with `guix graph -t
> module -b graphml guile`, and use e.g. networkx to analyze it.

Nice (we should add this trick under “Invoking guix graph” for instance;
we already mention xdot there.)

> You can compare with a compiled check-out of guix by just running the
> following in a `guix repl`:
>
> (use-modules (guix self) (guix monad-repl))
> ,run-in-store (guix-derivation (getcwd) "0.0-git" #:pull-version 1)
>
> which takes at most 5 seconds on my laptop.
>
> One idea I had was to move all the packages that are looked up in (guix
> self) to their own little bubble, to avoid having to load extra stuff.
> However, this is not currently possible because some of them do have
> non-trivial dependency graphs.  I've identified these problematic
> inputs: guile-avahi guile-ssh guile-git guile-gnutls guix-daemon (it
> pulls in all other dependencies itself) po4a graphviz

Yeah, I don’t think the idea of an independent subset of the module
graph is achievable, because sooner or later glibc starts depending on
Python, which starts depending on Rust, which pulls in Qt.  Compiling
with ‘-Wunused-modules’ might allow us to clean up the module graph a
bit though.

> What could be done about this?  Another solution would be to somehow
> build Guix without any of the dependencies and then add them in later,
> similar to what is done with build-aux/build-self.scm to be able to load
> (guix self) in the first place.  That seems quite complex though.

Most of the time is spent evaluating (gnu packages …) modules;
evaluation is pretty slow, and from what I remember of previous
profiling sessions, this is largely due to psyntax.  We should
concentrate on that.


Another approach: the resulting .drv is a deterministic result.
Wouldn’t it be nice if we could make the whole “Computing Guix
derivation” process a derivation itself?  That way it could be
substituted.

Nix introduced (or considered introducing) “recursive Nix”, whereby a
derivation build process can talk to the daemon “from the inside”,
creating additional derivations on the way.  This could be one
possibility, but it’s a lot of work with unclear outcomes.

In Guix, we can use the (guix …) modules inside the build process, and
we have almost everything we need to compute a set of derivations “in
the abstract” (we’d need a <store-connection> backend that, instead of
making RPCs, would store .drv and “sources” in memory or in a file).  So
we could create a derivation that returns let’s say a nar containing the
closure of the .drv (the “Guix derivation”), which we would then import.

There’s “just” one gotcha with this plan: grafts.  To compute grafts, we
potentially need to build things, and for that we’d need to actually
talk to the daemon.


Maybe there are other possibilities, such as adding a
“builtin:build-self” derivation builder in the daemon.

Food for thought!

Ludo’.