bug#30290: guix-daemon slows to a crawl when a substitute server is offline

[Maxim Cournoyer]

Hi!

zimoun <zimon.toutoune@gmail.com> writes:

> Hi Maxim,
>
> On Mon, 29 Jan 2018 at 22:07, Maxim Cournoyer <maxim.cournoyer@gmail.com> 
> wrote:
>> When a substitute server used by guix-daemon is offline, the daemon will
>> keep attempting to connect to it, even when it shouldn't need any data
>> (ran 'sudo guix system reconfigure my-config.scm' multiple times in a
>> row.
>>
>> With the disconnected server (bayfront in my case), that command would
>> take close to 8 minutes, with many system calls like:
>>
>> connect(14, {sa_family=AF_INET, sin_port=htons(443), 
>> sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
>>
>> which wasted 5 seconds each time.
>>
>> After removing this server from my substitute servers list, the same
>> operation (system reconfigure) is 8 times faster (1 minute).
>>
>> Suggestion: the daemon should stop trying to use the offline substitute
>> server after trying for X times, and print a warning about it.
>
> This looks like as a wishlist, right?  Do it make sense to include such
> feature to the recent discussions about the revamp of offloading,
> Cuirass, publish, etc.

To me it's an issue more than a feature request, especially in a build
farm setting; having a substitute machine down shouldn't cause a slow
down for as long as it's down!

I'm not sure if the recent offloading work that Mathieu did touched that
topic.  I'd need to test the scenario.  Perhaps a system test would be
useful.

Maxim