Re: Pinning package inputs using inferiors?

[Efraim Flashner]

On Fri, Oct 21, 2022 at 10:08:10PM +0100, Phil wrote:
> Thanks Simon - I've given an example below.
> 
> zimoun writes:
> 
> > For an example, see python-numpy and python-numpy-next in (gnu packages
> > python-xyz).
> 
> This was my original way of handling this but in what is perhaps a niche
> use of Guix by my department - it ultimately doesn't scale well, for our
> use-case.
> 
> Originally the department was small enough that there was only a handful
> of applications sharing one or two common in-house libraries.
> 
> As we've scaled-up we now have the situation where 3 or 4 common
> libraries are being used by say 10 applications.
> 
> We have rapid release schedules - and want to be able to release the
> common libraries on a weekly basis.  But the time to sign-off on a
> common library takes a few days per application, so it's not practical for
> every project to bump version every week - they have other priorities.
> 
> In an ideal world automated unit and regression testing would be
> comprehensive enough that we could move aggressively each week, but at
> least for now that's not practical given the complex nature of signing
> off the libraries and the applications which use the libraries.
> 
> So, ideally, what we'd like to do is just have each common library
> churn-out releases every week, and have the releases available in Guix,
> but without having an obligation on dependent applications to adopt
> these changes if they don't want to.
> 
> Note all libraries and applications share the same channel - one
> solution would be to have each library in their own channel, but this
> feels ugly to me.
> 
> Our solution (somewhat tricky to explain without a whiteboard -
> apologies!) is to co-locate the package definition of the common library
> in the common library repo itself - we call it something like
> .requirements.scm and it is naturally kept in lockstep with the code in
> that repo that the definition will build.  This is very different to
> traditional Guix where channels contain definitions separately in a
> different repo to the code those definitions describe how to build. 
> 
> We then have a job in our CI/CD system that allows us to give a tag on the
> common library repo, and the name of an application that uses the common
> library.
> 
> The job will copy the .requirements.scm into our channel inside a
> private module specific to the application that uses the common library.
> 
> The idea is that you can have many versions of .requirements.scm private
> to every application package definition that references it.
> 
> You could even read .requirements.scm using a function that clones the
> application repo on-the-fly rather than statically storing it in the
> channel - we haven't gone this far yet, as it makes the system even more
> complex to reason about.
> 
> This is basically the same idea as the python-numpy-next but allows for
> many versions of python-numpy to co-exist by keeping them all in private
> modules so they don't clash.
> 
> It's a cool idea and works pretty well, but requires us to augment Guix
> with a set of extra tools to lift and shift these private definitions
> around, which complicates our setup considerably.
> 
> It feels like wanting to make many versions of a library available at
> once isn't an unreasonable way to work at-scale.  However, it also feels
> like a departure from the philosophy of Guix to decentralise package
> definitions and to allow for a potentially large range of versions to
> co-exist in the same channel commit.
> 
> We could try to further integrate the idea into guix by writing new guix
> commands to support it - we're still working out the details ourselves,
> but if it works well we'd love to present it at a future Guix Days or
> similar!
> 
> In the meantime I was wondering if anyone else had a similar use-case
> for Guix and if they had tried something similar or different to handle
> many versions in an automated way in the same channel commit?
> 
> Apologies that's more than I was intending to write - but hopefully that
> makes some sense!  If it doesn't I can try to flesh out specific example?

Apologies for not slotting in the reply inline, I'm not sure exactly
where to put it.

This might be a good use for package transformations. Imagine the
following:

;;; Python-dep-1

(define-publid python-dep1-week1
  ...)

(define-publid python-dep1-week2
  ...)

(define-publid python-dep1-week3
  ...)

;;; Python-dep-2

(define-publid python-dep2-week1
  ...)

(define-publid python-dep2-week2
  ...)

(define-publid python-dep2-week3
  ...)

;;; Python package

(define my-python-package-base
  (name "my-python-package-base")
  ...
  (inputs
   (list python-dep1
         python-dep2)))

(define-public my-python-package
  (inherit my-python-package-base)
  (name "my-python-package")
  (inputs
   (modify-inputs (package-inputs python-package-base)
     (replace python-dep1 python-dep1-week3)
     (replace python-dep2 python-dep2-week2))))

or if you wanted to do it recursively

(define package-inputs-for-my-python-package
  (package-input-rewriting/spec
    `(("python-dep1" . ,(const python-dep1-week3))
      ("python-dep2" . ,(const python-dep2-week2)))))

(define-public python-package-with-correct-inputs
  (package
    (inherit (package-inputs-for-ython-package my-python-package-base))
    (name "my-python-package")))


Both ideas use 'placeholder packages' so that you can swap them out for
your actual dependency, the first one only for itself and the second one
working recursively. I use the second one myself at work to replace
tensorflow with a version built for the machine it's running on, and at
home to use some newer golang libraries.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

signature.asc
Description: PGP signature