[bug#36630] [PATCH] guix: parallelize building the manual-database

[Ludovic Courtès]

Hi Arne,

Arne Babenhauserheide <address@hidden> skribis:

> * guix/profiles.scm (manual-database): par-map over the entries.  This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O.  The order of the entries stays the same since write-mandb-database sorts
> them.

I would think the whole process is largely I/O-bound.  Did you try
measuring differences?

I picked the manual-database derivation returned for:

  guix environment --ad-hoc jupyter python-ipython python-ipykernel -n

(It has 3,046 entries.)

On a SSD and with a hot cache, on my 4-core laptop, I get 74s with
‘master’, and 53s with this patch.

However, it will definitely not scale linearly, so we should probably
cap at 2 or 4 threads.  WDYT?

Another issue with the patch is that the [n/total] counter does not grow
monotically now: it might temporally go backwards.  Consequently, at
-v1, users will see a progress bar that hesitates and occasionally goes
backward, which isn’t great.

This would need to fix it with a mutex-protected global counter.

All in all, I’m not sure this is worth the complexity.

WDYT?

Thanks,
Ludo’.