[bug#45253] [PATCH 0/6] Pipeline substitute integrity check, deduplication, and canonicalization

[Maxim Cournoyer]

Hey Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> I tested with substitutes that contain many files:
>>
>>   guix build pipewire@0.2 ffmpeg ungoogled-chromium vim-full \
>>     emacs-no-x emacs-no-x-toolkit
>>
>> On my laptop with an SSD, the wall-clock time is almost unchanged
>> when fetching lzip substitutes.  You can see that the throughput
>> displayed while downloading is slightly lower than before, which
>> is consistent because lzip downloads are CPU-bound¹, but this is
>> compensated by the lack of processing time between substitutes.
>> With gzip substitutes, I see a 10% speedup on the wall-clock time
>> on my laptop.
>
> Picture!  First the timechart with the current daemon (gzip
> substitutes, downloading from the LAN):
>
>
>
>
> Notice how guix-daemon is busy in between substitute downloads: that’s
> the time it takes to compute the nar hash of the store item, reset its
> timestamps, and deduplicate its files.
>
> Now the same operation after with this patch series:
>
>
>
>
> This time guix-daemon remains idle all along whereas ‘guix substitute’
> is almost 100% busy.  There’s no pause time between substitutes.
>
> Ludo’.

Very nice!  Thanks for this work!  I can't wait to try it.

Maxim