[bug#45018] [PATCH 0/6] Process and connection reuse for substitutions

[Ludovic Courtès]

Hi Guix!

The attached patches optimize substitute downloads by: (1) keeping
a single ‘guix substitute’ process for all the substitutes (instead
of respawning a new one each time a store item is substituted), and
(2) reusing connections to substitute servers in between substitutions.

(Note that ‘guix publish’ does not keep connections alive, but
on ci.guix.gnu.org we run nginx in front of ‘guix publish’, and nginx
keeps them alive.)

Anecdotally, the effect is a 10% improvement on the wall-clock time
of ‘guix build --sources=transitive vim’ on my laptop, from 29s to 26s.
Of course it cannot be much better since the rest of the time is spent
actually retrieving bits over the network.

Overall the impact depends on a number of factors.  My laptop has
an SSD and I have fiber-to-the-home (FFTH) with low latency: forking
and initiating a TLS connection to ci.guix.gnu.org are both quite cheap,
so I probably don’t benefit that much.  The impact may be more acute
on a low-end device.

In the ‘guix build --sources=transitive vim’ case, there are a few
large tarballs and several small ones.  The gain is in the lack of
a pause time in between small tarballs.  Also this case is quite
advantageous: there’s no decompression and no unpacking happening.

Anyway, I think it’s a welcome improvement, especially when
downloading lots of stuff such as during the initial system installation.
Plus, there are more deletions than insertions.  :-)

Thoughts?

Ludo’.

Ludovic Courtès (6):
  daemon: 'Agent' constructor takes a list of environment variables.
  daemon: Use 'Agent' to spawn 'guix substitute --query'.
  daemon: Factorize substituter agent spawning.
  daemon: Run 'guix substitute --substitute' as an agent.
  substitute: Cache and reuse connections while substituting.
  daemon: Raise an error if substituter doesn't send the expected hash.

 guix/http-client.scm        |  12 +-
 guix/progress.scm           |   8 +-
 guix/scripts/substitute.scm | 215 +++++++++++++++++++++++-------------
 nix/libstore/build.cc       | 166 ++++++++++++++--------------
 nix/libstore/local-store.cc | 170 +++++++---------------------
 nix/libstore/local-store.hh |  25 +----
 nix/libutil/util.cc         |   6 +-
 nix/libutil/util.hh         |   7 +-
 tests/substitute.scm        |  98 +++++++++-------
 9 files changed, 350 insertions(+), 357 deletions(-)

-- 
2.29.2