Re: Pinning package inputs using inferiors?

[Phil]

Thanks Simon - I've given an example below.

zimoun writes:

> For an example, see python-numpy and python-numpy-next in (gnu packages
> python-xyz).

This was my original way of handling this but in what is perhaps a niche
use of Guix by my department - it ultimately doesn't scale well, for our
use-case.

Originally the department was small enough that there was only a handful
of applications sharing one or two common in-house libraries.

As we've scaled-up we now have the situation where 3 or 4 common
libraries are being used by say 10 applications.

We have rapid release schedules - and want to be able to release the
common libraries on a weekly basis.  But the time to sign-off on a
common library takes a few days per application, so it's not practical for
every project to bump version every week - they have other priorities.

In an ideal world automated unit and regression testing would be
comprehensive enough that we could move aggressively each week, but at
least for now that's not practical given the complex nature of signing
off the libraries and the applications which use the libraries.

So, ideally, what we'd like to do is just have each common library
churn-out releases every week, and have the releases available in Guix,
but without having an obligation on dependent applications to adopt
these changes if they don't want to.

Note all libraries and applications share the same channel - one
solution would be to have each library in their own channel, but this
feels ugly to me.

Our solution (somewhat tricky to explain without a whiteboard -
apologies!) is to co-locate the package definition of the common library
in the common library repo itself - we call it something like
.requirements.scm and it is naturally kept in lockstep with the code in
that repo that the definition will build.  This is very different to
traditional Guix where channels contain definitions separately in a
different repo to the code those definitions describe how to build. 

We then have a job in our CI/CD system that allows us to give a tag on the
common library repo, and the name of an application that uses the common
library.

The job will copy the .requirements.scm into our channel inside a
private module specific to the application that uses the common library.

The idea is that you can have many versions of .requirements.scm private
to every application package definition that references it.

You could even read .requirements.scm using a function that clones the
application repo on-the-fly rather than statically storing it in the
channel - we haven't gone this far yet, as it makes the system even more
complex to reason about.

This is basically the same idea as the python-numpy-next but allows for
many versions of python-numpy to co-exist by keeping them all in private
modules so they don't clash.

It's a cool idea and works pretty well, but requires us to augment Guix
with a set of extra tools to lift and shift these private definitions
around, which complicates our setup considerably.

It feels like wanting to make many versions of a library available at
once isn't an unreasonable way to work at-scale.  However, it also feels
like a departure from the philosophy of Guix to decentralise package
definitions and to allow for a potentially large range of versions to
co-exist in the same channel commit.

We could try to further integrate the idea into guix by writing new guix
commands to support it - we're still working out the details ourselves,
but if it works well we'd love to present it at a future Guix Days or
similar!

In the meantime I was wondering if anyone else had a similar use-case
for Guix and if they had tried something similar or different to handle
many versions in an automated way in the same channel commit?

Apologies that's more than I was intending to write - but hopefully that
makes some sense!  If it doesn't I can try to flesh out specific example?