[bug#38649] [PATCH] Parallelize `guix package`

[Leo Prikler]

Hi Ludo’,

Am Dienstag, den 17.12.2019, 15:32 +0100 schrieb Ludovic Courtès:
> Hi Leo,
> 
> (Cc: Julien, who worked on this part.)
> 
> Leo Prikler <address@hidden> skribis:
> 
> > Yesterday I had an interesting conversation on IRC about the
> > behaviour
> > of multiple `guix package` processes running in parallel. 
> > Specifically, when two transactions target the same profile
> > (usually
> > /var/guix/profiles/per-user/$USER/guix-profile) at the same time,
> > one
> > of them will fail to claim the lock and abort.  0001 makes it so
> > that
> > the process waits for the lock.  0002 makes it so that packages
> > specified via -i can be built in parallel.
> 
> I actually like the current behavior, FWIW.  Julien came up with this
> locking mostly so that people do not inadvertently attempt to perform
> several operations concurrently.
Fair enough and that is an improvement over non-locking behaviour,
where one could spawn multiple profile generations from one, neither of
which is complete.  Perhaps my attempt at doing this in a somewhat
controlled manner is equally harmful, but I will still try my best
arguing for it, as I believe it can make a positive impact.

> The key word here is “inadvertently”: IMO, there’s no reason to run
> multiple ‘guix package’ on the same profile concurrently.  With a
> wait-for-lock policy, the result would be non-deterministic: you
> cannot
> tell which one of the two processes will complete first.
> 
> WDYT?
I think the current policy is wait-for-lock deferred to the user.  The
user has to let the first task complete before they can start the
second.  In this setup, the user can simply launch the setup and trust,
that it will complete later while taking into account the changes the
first one has made.

Let's talk about three classes of operations – installations, removals
and upgrades – and their interactions.  I will not take into account
roll-back, switch-generation and delete-generation, as it is
nonsensical to perform these in parallel to any other action.  Perhaps
we could check for their presence first and acquire the lock with no-
wait semantics in that case.

- any operation on different packages: Either succeeds first and the
other builds on the profile it generates. As there is no collision in
the packages themselves, there will be no harm.
- install same package twice: Either succeeds first, the other will be
a no-op.
- install vs. remove same package: Non-deterministic, but why would you
do that?
- install vs. upgrade same package: Upgrade will be a no-op in either
case.
- remove vs. upgrade same package: Upgrade may inadvertently upgrade
the old package if it happens to come first, but in the final package
it will be removed either way. 

Of course, any operation can also fail midway due to some step not
succeeding.  In that case it would be as if one had issued the other
command right after that, which may perhaps not be what one wanted to
do (assuming I install package A, and some guide suggests to also build
related, but not dependency-connected package B, so I end up installing
B without A).  However, such cases can easily be fixed by either
installing a fixed version of A later, using B on its own if it can be,
or rolling back.

Of course, both solutions are flawed in the way that they assume user
intent either way.  Perhaps a better one would be to let the user
specify whether they want to wait or not through a command line
parameter, using the current behaviour as the default approach.

WDYT?

Regards,
Leo