Re: Experimental nar-herder support for serving fixed output files by hash

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> The code isn't great, there's some difficulty in extracting the single
>>> file from the nar, but the biggest problem is a limitation in the guile
>>> fibers web server. Currently, responses have to be read in to memory,
>>> which is fine for we pages, but not great if you're trying to serve
>>> files which can be multiple gigabytes in size. This also means that the
>>> first byte of the response is available when all the bytes are
>>> available, so the download is slow to start.
>>
>> That, and in practice a cache (with some eviction mechanism) would be
>> necessary so nars don’t need to be extracted every time and so we can
>> use sendfile(2).
>
> I'd actually imagined that this would be used infrequently, but yeah, if
> the decompression does become a bottleneck, then some caching reverse
> proxy could help reduce that.

Relying on a proxy may be insufficient, because you still have incoming
requests that can trigger unbounded peaks of I/O and CPU usage, and
these requests may not be satisfied in time (the client may hang up
before the server is done processing the nar.)

>> IWBN to share as much code as possible with ‘guix publish’, which has
>> great test suite coverage and is being hammered every day.  Clearly the
>> bit about extracting nars is specific to the nar-herder though, so that
>> may prove difficult.
>
> I'm going to look at the Guile Fibers web server, hopefully that can be
> improved to support streaming responses, which would allow removing a
> lot of custom code from guix publish.

By “streaming responses”, do you mean pipelining?

How would that affect ‘guix publish’?

> There isn't all that much code to the nar-herder though, and most of
> waht is there is doing different things to guix publish, so I'm not sure
> there's all that much to share.
>
> What I was getting at here though, ignoring the implementation, was
> whether this is worthwhile to do? As in, is there benefit to having this
> and being able to extend the content addressed mirrors that Guix uses?

Having more content-addressed mirrors is worthwhile IMO, yes.

Having two different implementations of the same interfaces may not be
ideal, though, in terms of long-term maintenance cost.

Thanks,
Ludo’.