Another update on the Guix Data Service

[Christopher Baines]

Hey,

It's been a while since I send out an update about the Guix Data
Service, so here's a somewhat failed attempt at a short update about
recent progress.

There are formatted [1] and plain views of derivations [2]. The plain
view is constructed from the representation in the database, but it
should match what Guix produces.

1: 
http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/formatted
2: 
http://data.guix.gnu.org/gnu/store/0hpz4039vs2n514kbd3psh5dwl0dnqwg-guix-1.0.1-11.f38eabe.drv/plain

Channel news entries are recorded in the database, you can see all that
are present in a revision [3], but the comparison page should show new
news entries as well.

3: 
http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/news

There's now a page to compare two derivations [4], it's not a
particularly clear comparison yet, but can hopefully be improved in the
future. This is linked to from the derivation history pages [5].

4: 
http://data.guix.gnu.org/compare/derivation?base_derivation=/gnu/store/j3n1y6ak2hxs838lm6xx3akljf0xh5qd-guix-1.0.1-10.41b4b71.drv&target_derivation=/gnu/store/x6zncwgmmhf1lq7p805ixhaqfhwdxybk-guix-1.0.1-11.f38eabe.drv
5: 
http://data.guix.gnu.org/repository/1/branch/master/package/guix/derivation-history

There's support for receiving build information from build servers, but
this hasn't been used yet. I'm slowly looking at getting support for
using this in to Cuirass [6]

6: https://lists.gnu.org/archive/html/guix-devel/2019-10/msg00467.html

Information about nars (substitutes) can now be stored ([7] for
example). This data also shows up on a page for derivation outputs for a
revision ([8] for example).

7: 
http://data.guix.gnu.org/gnu/store/001j0wk7srm7jlnd9lai7p1yzg4bdl3k-dunst-1.4.1
8: 
http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/derivation-outputs

There's a page for builds for a revision [9]. There's also a page for
individual builds [10].

9: 
http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/builds
10: 
http://data.guix.gnu.org/build-server/2/build?derivation_file_name=/gnu/store/yc0pb7c0c198nb68c5wha41brzvrls01-k3b-19.08.3.drv

The derivation outputs page for a revision shows information about
outputs matching across build servers, but there's also a page about
package reproducibility for a revision [11].

11: 
http://data.guix.gnu.org/revision/06c8e87224decebc0a3d5bdc7d2ca13cb2b08299/package-reproducibility

Provide narinfo and nar files for derivations, and the required source
files ([12] for example). This means the Guix Data Service can be used
as a substitute server for derivations, for example [13].

12: http://data.guix.gnu.org/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1.narinfo
13: guix build --substitute-urls="http://data.guix.gnu.org 
https://ci.guix.gnu.org"; 
/gnu/store/l9kznkzkwj1hilbx39nrnlkpl1s6rxz1-hello-2.10.drv

Fix generating derivations for foreign architectures. There was an issue
where only the x86_64-linux and i686-linux derivations were properly
loaded, but now that's been fixed.

I added support for loading mbox files a while ago, but I've only now
got around to testing a few older revisions, and now I've loaded the
mbox files for the first 3 months from 2019 so eventually those will be
loaded in.


A few things above were influenced by thinking about reproducible builds
in the run up, during and after the reproducible builds summit at the
start of December. I'm excited that there's some support in the Guix
Data Service for comparing builds across build farms, but there's still
a lot to do in terms of getting every derivation built by at least 2
separate build farms, and then getting that data promptly in to the Guix
Data Service for comparison.

Also at the reproducible builds summit, Ludovic made me aware that you
can tell Guix to build a derivation, and if it doesn't have it locally,
it will attempt to download it from substitute servers. I've now got the
Guix Data Service setup to act as a substitute server for derivations,
which while on it's own isn't particularly exciting (as build farms
already can provide substitutes for derivations), it's now possible to
get the information about derivations within the Guix Data Service, out
of the Guix Data Service in a usable manor. This might have applications
in the future, like speeding up guix pull by providing the derivation
that corresponds to a revision, or providing derivations for a build
farm to build.

Like the example of providing substitutes for derivations, quite a lot
of this work isn't directly impactful (at least yet), but I'm hoping
it's laid the foundations for some more directly impactful stuff.

In particular, I'm still thinking about;

 - Automated code review for Guix patches, which I was working on when I
   started working on the Guix Data Service, and most of the
   functionality can be beneficial there.

 - The Guix Weekly News project I was working on recently uses data from
   the Guix Data Service, and that's what motivated storing the channel
   news entries.

 - Trust in builds, once the data about package build reproducibility is
   more complete, hopefully that will be informative and show where
   build reproducibility issues can be fixed.

 - Improving the speed in which substitutes are made available.

 - Explaining the data that underpins Guix to a technical audience, like
   derivations and how they work.

Do let me know if you have any thoughts or questions, I'm also going to
be around at the Guix Days at the end of January.

Thanks,

Chris

signature.asc
Description: PGP signature