Guix Data Service - September update

[Christopher Baines]

Hey,

I think I sent out the last update about the Guix Data Service back in May
[1], and quite a few things have changed since then. This is a summary of
changes since then, and a list of things that I'm interested in looking at
next.

1: https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00332.html
   More progress with the Guix Data Service (17th of May)

I ran out of disk space on the server I'd been using to run the Guix Data
Service [2] so I removed it. Thanks to a kind donation of a machine from
UNIMI-DI through Giovanni Biscuolo, I deployed the Guix Data Service to
milano-guix-1, which has a lot more resources. It was down recently due to a
disk failure, but it's back online now [3].

2: https://prototype-guix-data-service.cbaines.net/ (no longer used)
3: http://milano-guix-1.di.unimi.it:8765/

It's not currently running on the standard ports, as I haven't got around to
setting up NGinx yet, but that's something I'm looking to work on soon. Quite
a few things have changed (and hopefully improved) in the last few months,
I've tried to give a summary below, but it's probably easier just to have a
look at it running on milano-guix-1 [3].

There have been many changes to the user interface, the index page has changed
to show a list of branches (rather than some revisions and jobs), there are
more links between pages, and some pages now link to cgit where useful. The
404 pages have been improved and cache headers are now set as well.

For processing jobs, the records in the database used to be deleted when the
job was completed, but now the records are kept and there's a page showing
jobs [4]. Also, the code now supports processing jobs without container
support for inferiors in Guix, and can process jobs in parallel, prioritising
the latest revision for each branch. Separate processes are used for each job
to allow concurrency, as well as improving memory management as those
processes exit when the job is finished. The log handling for jobs is also
more efficient.

4: http://milano-guix-1.di.unimi.it:8765/jobs

In terms of the Guix service, Sqitch is integrated in to the guix-data-service
script that provides the web server, so database migrations can be
automatically run on startup. There's also an option to create a pid file,
which is useful as it prevents the jobs process from starting until migrations
have been applied to the database.

I investigated why the comparison function was broken, and it turned out that
some unique constraints didn't work as intended in the case of columns with
NULL values, and the queries around inserting data failed in a similar
way. This is now handled properly, and there are migrations to remove the
duplicate values from the database. This was breaking some of the comparison
functionality.

Most recently, lint warnings for lint checkers that don't require network
access are stored in the database. The warnings are displayed on the revision
page, and included on the compare page (for example [5]). This follows on from
the changes that I started talking about here [6].

5: 
http://milano-guix-1.di.unimi.it:8765/compare?base_commit=e1e3fe08480868f960eea3ec1584c0c12b022e25&target_commit=067ea2989fce98f3f3f115534e2e685cfc681039
6: https://lists.gnu.org/archive/html/guix-devel/2019-05/msg00127.html
   Linting, and how to get the information in to the Guix Data Serivce (6th May)

Other smaller things:

 - There are pages for the latest processed revision for a branch (e.g. [7])

 - There's less code duplication for the code relating to inserting new data
   in to the database.

 - Names are with each database connection, so it's easier to see what each
   connection is doing

 - glibc-locales from the inferior Guix is used when loading data, which fixes
   some locale issues

 - I hacked some better NULL value support on top of guile-squee [8]`

 - I started changing the code to handle data in the natural type (e.g. number
   for numbers), rather than using strings. This worked for a while as squee
   always returned and expected strings, but to provide more adaptable code
   for working with the database, being able to use the type information for
   each value is really useful.

7: 
http://milano-guix-1.di.unimi.it:8765/repository/1/branch/master/latest-processed-revision
8: 
https://git.cbaines.net/guix/data-service/commit/?id=14419422008cc1ba42dea5ef90e6fb2762633064

In terms of what's next:

 - I've started writing a proposal for the upcoming Outreachy round relating
   to internationalisation in the Guix Data Service

 - I want to get back to making progress on automating code review for Guix
   patches, this was one of the main motivations for getting lint warnings in
   the database and on to the comparison page

 - I want to provide public dumps of the database from milano-guix-1, as well
   as a small extract of that database. I think restoring a database locally
   is a good way to get data for local development

 - Relating to the Outreachy proposal but also generally, I want to write some
   documentation on how to get the Guix Data Service running locally

 - Currently the Git repository is on my personal Git server, and there are
   discussions about moving it to Savannah

 - The Guix package and service definitions haven't been merged, so I want to
   look at that once the location for the Git repository is sorted out

Do let me know if you have any comments, questions, or also interested in
this,

Chris

signature.asc
Description: PGP signature