Re: Experimental nar-herder support for serving fixed output files by hash

[Ludovic Courtès]

Hello,

Christopher Baines <mail@cbaines.net> skribis:

> With the nar-herder, the implementation is a little trickier. Since the
> nar-herder manages a collection of nars, rather than serving things from
> the store, it might have the file being requested but it's inside a
> probably compressed nar file. So, to respond to these requests, the
> nar-herder has to take the relevant nar file and then read the file out
> of it. I've now got an initial implementation of this:
>
>   
> https://git.cbaines.net/guix/nar-herder/commit/?id=042f49e5fb52ea844ed5d29c17b26fbc8ad49f0e

Interesting.

> The code isn't great, there's some difficulty in extracting the single
> file from the nar, but the biggest problem is a limitation in the guile
> fibers web server. Currently, responses have to be read in to memory,
> which is fine for we pages, but not great if you're trying to serve
> files which can be multiple gigabytes in size. This also means that the
> first byte of the response is available when all the bytes are
> available, so the download is slow to start.

That, and in practice a cache (with some eviction mechanism) would be
necessary so nars don’t need to be extracted every time and so we can
use sendfile(2).

> With all of that said though, it does seem to work. For testing, I've
> enabled it on bishan, which serves the bordeaux.guix.gnu.org collection
> of nars. It only has IPv6 connectivity, so you'll only be able to try
> this out if you've got an IPv6 support locally:
>
>   
> https://bishan.guix.gnu.org/file/0ad-0.0.25b-alpha.tar.xz/sha256/1p9fa8f7sjb9c5wl3mawzyfqvgr614kdkhrj2k4db9vkyisws3fp

Nice!

> In terms of next steps, there's some things to do with improving the
> implementation, but it would be good to hear if this is actually
> worthwile?

IWBN to share as much code as possible with ‘guix publish’, which has
great test suite coverage and is being hammered every day.  Clearly the
bit about extracting nars is specific to the nar-herder though, so that
may prove difficult.

Thoughts?

Ludo’.