Re: Disarchive update

[Ludovic Courtès]

Hello Guix!

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

> This job is disassembling all the .tar.gz files packages refer to, using
> the recently-added ‘etc/disarchive-manifest.scm’ file:
>
>   https://ci.guix.gnu.org/jobset/disarchive

[...]

> A plan we can already deploy would be:
>
>   1. Add the disarchive.guix.gnu.org DNS entry, pointing to berlin.

Done:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=df9e9b7f51abceb5999aabc9a7b71396600cffa4
  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=12195160432871b80d0e1eac996a9aa7d8500697

Sample URLs:

  
https://disarchive.guix.gnu.org/sha256/53cf3e14c71f3a149f29d13a0da64120b3c1d3334fba39c4af3e520be053982a
  
https://disarchive.guix.gnu.org/sha256/39052f59ff474a4a69cefc25cf3caf8429400889deba010ee6403ca188f8b311
  
https://disarchive.guix.gnu.org/sha256/03a71d53055bd9ec528d55e07afaf15c09dec9856cba734904bfd05acbc6cf12

Aren’t those Disarchive sexps really cute? :-)

>   2. On berlin, add an mcron job that periodically copies the output of
>      the latest “disarchive-collection” build to a directory, say
>      /srv/disarchive.  Thus, the database would accumulate tarball
>      metadata over time.

First, there’s a script to populate the database; it copies files from
the latest successful “disarchive-collection” build to a specified
directory, gzipping them on their way.  It’s atomic, so the directory in
question can be directly served by nginx or similar:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=fb83b3d8de189c6d6c33c4cdc2ebabf6eae1463e

If you want to try it at home, just run:

  ./sync-disarchive-db.scm /tmp/db

It’s pretty fast!  The output is only 70 MiB, now that individual files
are gzipped.

Then there’s the mcron job that runs it once a day on berlin:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=27dc74fbe33a9d929b37994e825dc202385f87c0

We could run it as well on bayfront so we have a backup.

>   3. Add an nginx route so that /srv/disarchive is served at
>      https://disarchive.guix.gnu.org.

Done here:

  
https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=9ffb2db81a2fbee67b99c76217be874ec0fd6bde

>   4. Add disarchive.guix.gnu.org to (guix download).

Done:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f9a506aa6a5aaeb2c06c97d5b663d01d2103db69

As I was once again modifying files by hand to test the download
fallback mechanisms, I figured we could just as well add a variable to
enable testing, which is what I did here:

  
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c4a7aa82e25503133a1bd33148d17968c899a5f5

So you can do, say:

  GUIX_DOWNLOAD_FALLBACK_TEST=disarchive-mirrors guix build -S r-ebimage --check

or:

  GUIX_DOWNLOAD_FALLBACK_TEST=content-addressed-mirrors guix build -S r-ebimage 
--check

to check whether these fallback mechanisms work as expected.  (They do,
but I’ll update the ‘guix’ package because the current one has a bug
that breaks the Disarchive/SWH fallback.)

I think we’re making progress!  :-)

Thanks,
Ludo’.