[bug#47336] Disarchive as a fallback for downloads

[zimoun]

Hi Timothy,

(CC Mathieu to advice if it could be a feature of Cuirass.)


On Tue, 23 Mar 2021 at 00:42, Timothy Sample <samplet@ngyro.com> wrote:

> This patch series adds Disarchive assembly (backed by SWH lookup) as a
> fallback for downloads.

Awesome!


> You also need to make sure that regular downloads are unavailable.  I do
> this by adjusting the “try” loop at the end of “url-fetch” in
> “guix/build/download.scm”.  I replace the usual list of URLs with ‘()’:
>
>     (let try ((uri (append uri content-addressed-uris)))
>       (match '() ; uri
>         ...))
>
> Now you can ask Guix for a recent .tar.gz source package:
>
>     $ ./pre-inst-env guix build --no-substitutes -S python-httpretty

Neat!  Now, there is a way to easily check the coverage, right?  Since
SWH is ingesting the tarball using <http://guix.gnu.org/sources.json>,
there is now a mean to report what Guix is able to rebuild.

>     Checking httpretty-1.0.5 digest... ok

What happens if it is not ok?

>     Assembling the tarball httpretty-1.0.5.tar
>     Checking httpretty-1.0.5.tar digest... ok
>     Assembling the Gzip file httpretty-1.0.5.tar.gz
>     Checking httpretty-1.0.5.tar.gz digest... ok
>     Copying result to 
> /gnu/store/kbcnm57y2q1jvhvd8zw1g5vdiwlv19y9-httpretty-1.0.5.tar.gz

Where is the assembly done?  In /tmp/, right?

>     successfully built 
> /gnu/store/k0b3c7kgzyn1nlyhx192pcbcgbfnhnwa-httpretty-1.0.5.tar.gz.drv

Just to be sure, when does Guix check the integrity checksum?  I mean,
does Guix check the checksum after ’disassemble’ re-assembled the source?


> First, it looks up the metadata on my server.  This is fine for a demo,
> but not what we want forever.  The patch series supports adding
> several

As we talked before, how does the database scale?  Do you have some
numbers for the current demo?  In order to try to extrapolate what does
it mean for a server to «store the metadata».

> mirrors for looking up the metadata.  In the past, we talked about
> putting everything on one or a few of the big Git hosting platforms like
> GitHub or Gitlab.  That way, it would be easily picked up by SWH and
> archived “forever”.  Right now, I have Cuirass set up to build the
> metadata, and a little script that moves it from the build server to my
> Web server.  It would be simple enough to adjust that script to push it
> to a remote Git repo.  (Of course, the next step is to move this setup
> to Guix infrastructure.)  Thoughts?

Maybe this database could be a package, say “guix-tarball-db”, updated
in agreement with the package “guix”.  The source of this
“guix-tarball-db” would be a remote big Git hosting platforms like
GitHub or whatever and not stored on Guix infrastructure, or maybe
stored on Guix infra.

Regularly, i.e., when the package “guix” is updated, in the same time,
the package “guix-tarball-db” is updated too.  The “guix lint -c
archival” sends the saving request to SWH.  Even if this saving request
should be automated soon. :-)

Then if Cuirass would have a feature to disassemble and update the Git
repo.

Last, a service should run as your demo.  But for long-term, this
service could disappear––assuming SWH not :-).  Therefore, we could
imagine installing “guix-tarball-db” then tweak some parameters of the
guix-daemon and “guix build <foo>”.  Both installing and building would
fetch from SWH if both upstream disappear.

Or this “guix-tarball-db” should not be a plain package but only an
input as origin for the package “guix”.


> Hopefully everything else is more-or-less fine.  :)

Thanks!  That’s awesome!


Cheers,
simon