Re: Publishing with Lzip

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <address@hidden> skribis:

> I've just sent a patch of a first draft for the Lzip bindings:
>
>   http://debbugs.gnu.org/cgi/bugreport.cgi?bug=34807
>
> Switching from Gzip (Zlib) to Lzip would save up to some 50% disk usage
> for substitutes which would greatly benefit the servers and help
> reducing the bandwidth usage.

Woohoo, awesome!

> A few things are missing in the patch and I would probably need some help:
>
> - The %liblz variable in guix/config.scm.in is not set dynamically by
>   autoconf.
>   This is because lzlib does not have a pkg-config entry.  I don't know
>   much about autoconf, so if someone knows how to do this properly...

I can give it a spin.

> - I'm still a bit confused about how the API is supposed to work.  There
>   is a lz-(de)compress-finish function which is supposed to be called
>   according to the examples but the manual does not really says why.  If
>   I call it, the tests fail.
>
> - I'm also not 100% I'm doing the right thing with the encoder/decoder:
>   should we write everything first, then read as much as we can?  Or
>   chain write-read calls like in bbexample.c?
>
> - I'm not 100% sure either that the terminating chunk will always be
>   compressed / decompressed.  (It works in the test though.)  I don't
>   really understand how Lzlib handles that part.

It’d be great to gain some confidence about these things.  :-) I haven’t
looked at the patch yet, but if you haven’t done it yet, I’d suggest
having tests like the one in tests/zlib.scm (compress and decompress a
bytevector of a random size with random contents), and you could
possibly perform more “stressful” tests manually as well (try to
compress/decompress tarballs, etc.)

I’d also recommend to re-read the API doc in the headers or whatever.
IME these APIs are very tricky to use and one has to pay attention to
every small detail.

> - How can I map between C enums and Guile with dynamic FFI?  This would
>   be useful to have improve error messages.

According to the C standard an enum is an ‘int’.  So mapping them is
just a matter of producing/consuming ints.  The values of the enum start
from 0 and are incremented by 1 from then on, unless specific values are
provided.

> - lzlib.scm is not used for publishing in the patch.  Will do that
>   later.  What are the strategies for transitioning from .gz to .lz?  I
>   suggest the following:
>
>   - On publishing, replace .gz with .lz compression.
>   - When extracting, check the type and call the appropriate format 
> decompressor.

The strategy I think would be to first have the complete tool chain in
Guix, that is support in ‘guix substitute’ and ‘guix publish’.

We won’t be able to change our servers to produce lzip overnight,
because old instances of ‘guix substitute’ wouldn’t be able to consume
it.

Perhaps one option would be to allow ‘guix publish’ to produce both gzip
and lzip, which we’d use during some transition period.  The difficulty
would be that narinfos currently provide just one URL for the nar, so
we’d need to either provide several URLs, or provide the right URL based
on some appropriate HTTP request header.  Let’s focus on the rest for
now.  :-)

Thank you!

Ludo’.