[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
branch master updated: website: Add draft about packages as channels.
|
From: |
Ludovic Courtès |
|
Subject: |
branch master updated: website: Add draft about packages as channels. |
|
Date: |
Thu, 01 Jun 2023 11:19:51 -0400 |
This is an automated email from the git hooks/post-receive script.
civodul pushed a commit to branch master
in repository guix-artwork.
The following commit(s) were added to refs/heads/master by this push:
new dbc448b website: Add draft about packages as channels.
dbc448b is described below
commit dbc448b94a25564a7bb5d139243a5ea7845529dd
Author: Ludovic Courtès <ludo@gnu.org>
AuthorDate: Thu Jun 1 17:16:02 2023 +0200
website: Add draft about packages as channels.
* website/drafts/package-channel.md: New file.
---
website/drafts/package-channel.md | 584 ++++++++++++++++++++++++++++++++++++++
1 file changed, 584 insertions(+)
diff --git a/website/drafts/package-channel.md
b/website/drafts/package-channel.md
new file mode 100644
index 0000000..aa42c05
--- /dev/null
+++ b/website/drafts/package-channel.md
@@ -0,0 +1,584 @@
+title: From development environments to continuous integration—the ultimate
guide to software development with Guix
+alttitle: Continuous integration, continuous delivery, and development
environments, all at once
+author: Ludovic Courtès
+tags: Software development, Continuous integration, Cuirass
+date: 2023-06-02 15:00:00
+---
+
+Guix is a handy tool for developers; [`guix
+shell`](https://guix.gnu.org/manual/en/html_node/Invoking-guix-shell.html),
+in particular, gives a standalone development environment for your
+package, no matter what language(s) it’s written in. To benefit from
+it, you have to initially write a package definition and have it either
+in Guix proper, in a channel, or directly upstream as a `guix.scm` file.
+This last option is appealing: all developers have to do to get set up
+is clone the project's repository and run `guix shell`, with no
+arguments—we looked at the rationale for `guix shell` in [an earlier
+missive](https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/).
+
+Development needs go beyond development environments though. How can
+developers perform continuous integration of their code in Guix build
+environments? How can they deliver their code straight to adventurous
+users? This post describes a set of files and conventions developers
+can follow in their repository to set up Guix-based development
+environments, continuous integration, and continuous delivery—all at
+once.
+
+# Getting started
+
+How do we go about “Guixifying” a repository? The first step, as we’ve
+seen, will be to add a `guix.scm` at the root of the repository we’re
+interested in. We’ll take [Guile](https://www.gnu.org/software/guile)
+as an example in this post: it’s written in Scheme (mostly) and C, and
+has a number of dependencies—a C compilation tool chain, C libraries,
+Autoconf and its friends, LaTeX, and so on. The resulting `guix.scm` is
+looks like the usual [package
+definition](https://guix.gnu.org/manual/en/html_node/Defining-Packages.html),
+just without the `define-public` bit:
+
+```scheme
+;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.
+
+(use-modules (guix)
+ (guix build-system gnu)
+ ((guix licenses) #:prefix license:)
+ (gnu packages autotools)
+ (gnu packages bash)
+ (gnu packages bdw-gc)
+ (gnu packages compression)
+ (gnu packages flex)
+ (gnu packages gdb)
+ (gnu packages gettext)
+ (gnu packages gperf)
+ (gnu packages libffi)
+ (gnu packages libunistring)
+ (gnu packages linux)
+ (gnu packages pkg-config)
+ (gnu packages readline)
+ (gnu packages tex)
+ (gnu packages texinfo)
+ (gnu packages version-control))
+
+(package
+ (name "guile")
+ (version "3.0.99-git") ;funky version number
+ (source #f) ;no source
+ (build-system gnu-build-system)
+ (native-inputs
+ (append (list autoconf
+ automake
+ libtool
+ gnu-gettext
+ flex
+ texinfo
+ texlive-base ;for "make pdf"
+ texlive-epsf
+ gperf
+ git
+ gdb
+ strace
+ readline
+ lzip
+ pkg-config)
+
+ ;; When cross-compiling, a native version of Guile itself is
+ ;; needed.
+ (if (%current-target-system)
+ (list this-package)
+ '())))
+ (inputs
+ (list libffi bash-minimal))
+ (propagated-inputs
+ (list libunistring libgc))
+
+ (native-search-paths
+ (list (search-path-specification
+ (variable "GUILE_LOAD_PATH")
+ (files '("share/guile/site/3.0")))
+ (search-path-specification
+ (variable "GUILE_LOAD_COMPILED_PATH")
+ (files '("lib/guile/3.0/site-ccache")))))
+ (synopsis "Scheme implementation intended especially for extensions")
+ (description
+ "Guile is the GNU Ubiquitous Intelligent Language for Extensions,
+and it's actually a full-blown Scheme implementation!")
+ (home-page "https://www.gnu.org/software/guile/";)
+ (license license:lgpl3+))
+```
+
+Quite a bit of boilerplate, but now someone who’d like to hack on Guile
+just needs to run:
+
+```
+guix shell
+```
+
+That gives them a shell containing all the dependencies of Guile: those
+listed above, but also _implicit dependencies_ such as the GCC tool
+chain, GNU Make, sed, grep, and so on. The chef’s recommendation:
+
+```
+guix shell -CP
+```
+
+That gives a shell in an isolated container, and all the dependencies
+show up in `$HOME/.guix-profile`, which plays well with caches such as
+[`config.cache`](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Cache-Files.html)
+and absolute file names recorded in generated `Makefile`s and the likes.
+The fact that the shell runs in a container brings peace of mind:
+nothing but the current directory and Guile’s dependencies is visible
+inside the container; nothing from the system can possibly interfere
+with your development.
+
+# Level 1: Building with Guix
+
+Now that we have a package definition, why not also take advantage of it
+so we can build Guile with Guix? We had left the `source` field empty,
+because `guix shell` above only cares about the *inputs* of our
+package—so it can set up the development environment—not about the
+package itself.
+
+To build the package with Guix, we’ll need to fill out the `source`
+field, along these lines:
+
+```scheme
+(use-modules (guix)
+ (guix git-download) ;for ‘git-predicate’
+ …)
+
+(define vcs-file?
+ ;; Return true if the given file is under version control.
+ (or (git-predicate (current-source-directory))
+ (const #t))) ;not in a Git checkout
+
+(package
+ (name "guile")
+ (version "3.0.99-git") ;funky version number
+ (source (local-file "." "guile-checkout"
+ #:recursive? #t
+ #:select? vcs-file?))
+ …)
+```
+
+Here’s what we changed:
+
+ 1. We add `(guix git-download)` to our set of imported modules, so we
+ can use its `git-predicate` procedure.
+ 2. We defined `vcs-file?` as a procedure that returns true when passed
+ a file that is under version control. For good measure, we add a
+ fallback case for when we’re not in a Git checkout: always return
+ true.
+ 3. We set `source` to a
+
[`local-file`](https://guix.gnu.org/manual/devel/en/html_node/G_002dExpressions.html#index-local_002dfile)—a
+ recursive copy of the current directory (`"."`), limited to files
+ under version control (the `#:select?` bit).
+
+From there on, our `guix.scm` file serves a second purpose: it lets us
+build the software with Guix. The whole point of building with Guix is
+that it’s a “clean” build—you can be sure nothing from your working tree
+or system interferes with the build result—and it lets you test a
+variety of things. First, you can do a plain native build:
+
+```
+guix build -f guix.scm
+```
+
+But you can also build for another system (possibly after setting up
+[offloading](https://guix.gnu.org/manual/devel/en/html_node/Daemon-Offload-Setup.html)
+or [transparent
+emulation](https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-emulation)):
+
+```
+guix build -f guix.scm -s aarch64-linux -s riscv64-linux
+```
+
+… or cross-compile:
+
+```
+guix build -f guix.scm --target=x86_64-w64-mingw32
+```
+
+You can also use [package transformation
+options](https://guix.gnu.org/manual/en/html_node/Package-Transformation-Options.html)
+to test package variants:
+
+```
+# What if we built with Clang instead of GCC?
+guix build -f guix.scm \
+ --with-c-toolchain=guile@3.0.99-git=clang-toolchain
+
+# What about that under-tested configure flag?
+guix build -f guix.scm \
+ --with-configure-flag=guile@3.0.99-git=--disable-networking
+```
+
+Handy!
+
+# Level 2: The repository as a channel
+
+We now have a Git repository containing (among other things) a package
+definition. Can’t we turn it into a
+[*channel*](https://guix.gnu.org/manual/devel/en/html_node/Channels.html)?
+After all, channels are designed to ship package definitions to users,
+and that’s exactly what we’re doing with our `guix.scm`. Granted, our
+Git repository is one package definition lost in a sea of code—in this
+case, Guile—, but still.
+
+Turns out we can indeed turn into a channel, but with one caveat: we
+must create a separate directory for the `.scm` file(s) of our channel
+so that `guix pull` doesn’t end loading unrelated `.scm` files when
+someone pulls the channel—and in Guile, there are lots of them! So
+we’ll start like this, keeping a top-level `guix.scm` symlink for the
+sake of `guix shell`:
+
+```
+mkdir .guix
+mv guix.scm .guix/guile-package.scm
+ln -s .guix/guile-package.scm guix.scm
+```
+
+(In Guile we actually used `build-aux/guix` instead of `.guix`, but the
+latter is probably clearer.) To make it usable as part of a channel, we
+need to turn our `guix.scm` file into a
+[module](https://guix.gnu.org/manual/devel/en/html_node/Package-Modules.html):
+we do that by changing the `use-modules` form at the top to a
+`define-module` form. We also need to actually *export* a package
+variable, with `define-public`, while still returning the package value
+at the end of the file so we can still use `guix shell` and `guix build
+-f guix.scm`. The end result looks like this (not repeating things that
+haven’t changed):
+
+```scheme
+(define-module (guile-package)
+ #:use-module (guix)
+ #:use-module (guix git-download) ;for ‘git-predicate’
+ …)
+
+(define-public guile
+ (package
+ (name "guile")
+ (version "3.0.99-git") ;funky version number
+ …))
+
+;; Return the package object define above at the end of the module.
+guile
+```
+
+We need one last thing: a [`.guix-channel`
+file](https://guix.gnu.org/manual/devel/en/html_node/Package-Modules-in-a-Sub_002ddirectory.html)
+so Guix knows where to look for package modules in our repository:
+
+```scheme
+;; This file lets us present this repo as a Guix channel.
+
+(channel
+ (version 0)
+ (directory ".guix")) ;look for package modules under .guix/
+```
+
+To recap, we now have these files:
+
+```
+.
+├── .guix-channel
+├── guix.scm → .guix/guile-package.scm
+└── .guix
+ └── guile-package.scm
+```
+
+And that’s it: we have a channel! (We could do better and support
+[*channel
+authentication*](https://guix.gnu.org/manual/en/html_node/Specifying-Channel-Authorizations.html)
+so users know they’re pulling genuine code. We’ll spare you the details
+here but it’s worth considering!) Users can pull from this channel by
+[adding it to
+`~/.config/guix/channels.scm`](https://guix.gnu.org/manual/devel/en/html_node/Specifying-Additional-Channels.html),
+along these lines:
+
+```
+(append (list (channel
+ (name 'guile)
+ (url "https://git.savannah.gnu.org/git/guile.git";)
+ (branch "main")))
+ %default-channels)
+```
+
+After running `guix pull`, we can see the new package:
+
+```
+$ guix describe
+Generation 264 May 26 2023 16:00:35 (current)
+ guile 36fd2b4
+ repository URL: https://git.savannah.gnu.org/git/guile.git
+ branch: main
+ commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
+ guix c5bc698
+ repository URL: https://git.savannah.gnu.org/git/guix.git
+ commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
+$ guix package -A ^guile$
+guile 3.0.99-git out,debug guile-package.scm:51:4
+guile 3.0.9 out,debug gnu/packages/guile.scm:317:2
+guile 2.2.7 out,debug gnu/packages/guile.scm:258:2
+guile 2.2.4 out,debug gnu/packages/guile.scm:304:2
+guile 2.0.14 out,debug gnu/packages/guile.scm:148:2
+guile 1.8.8 out gnu/packages/guile.scm:77:2
+$ guix build guile@3.0.99-git
+[…]
+/gnu/store/axnzbl89yz7ld78bmx72vpqp802dwsar-guile-3.0.99-git-debug
+/gnu/store/r34gsij7f0glg2fbakcmmk0zn4v62s5w-guile-3.0.99-git
+```
+
+That’s how, as a developer, you get your software delivered directly in
+the hands of users! No intermediaries, yet no loss of transparency and
+provenance tracking.
+
+With that in place, it also becomes trivial for anyone to create Docker
+images, Deb/RPM packages, or plain tarball of the software with [`guix
+pack`](https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html):
+
+```
+# How about a Docker image of our Guile snapshot?
+guix pack -f docker -S /bin=bin guile@3.0.99-git
+
+# And a relocatable RPM?
+guix pack -f rpm -R -S /bin=bin guile@3.0.99-git
+```
+
+# Bonus: Package variants
+
+We now have an actual channel, but it contains only one package. While
+we’re at it, we can [define package
+variants](https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html)
+in our `guile-package.scm` file, variants that we want to be able to
+test as Guile developers—similar to what we did above with
+transformation options. We can add them like so:
+
+```scheme
+;; This is the ‘.guix/guile-package.scm’ file.
+
+(define-module (guile-package)
+ …)
+
+(define-public guile
+ …)
+
+(define (package-with-configure-flags p flags)
+ "Return P with FLAGS as addition 'configure' flags."
+ (package/inherit p
+ (arguments
+ (substitute-keyword-arguments (package-arguments p)
+ ((#:configure-flags original-flags #~'())
+ #~(append #$original-flags #$flags))))))
+
+(define-public guile-without-threads
+ (package
+ (inherit (package-with-configure-flags guile
+ #~'("--without-threads")))
+ (name "guile-without-threads")))
+
+(define-public guile-without-networking
+ (package
+ (inherit (package-with-configure-flags guile
+ #~'("--disable-networking")))
+ (name "guile-without-networking")))
+
+
+;; Return the package object define above at the end of the module.
+guile
+```
+
+We can build these variants as regular packages once we’ve pulled the
+channel, or, if we have a checkout of Guile, we can run a command like
+this one from the top level:
+
+```
+guix build -L $PWD/.guix guile-without-threads
+```
+
+
+# Level 3: Setting up continuous integration
+
+This channel becomes even more interesting once we set up [continuous
+integration](https://en.wikipedia.org/wiki/Continuous_integration) (CI).
+There are several ways to do that.
+
+You can use one of the mainstream continuous integration tools, such as
+GitLab-CI. To do that, you need to make sure you run jobs into a Docker
+image or virtual machine that has Guix installed. If we were to do that
+in the case of Guile, we’d have a job that runs a shell command like
+this one:
+
+```
+guix build -L $PWD/.guix guile@3.0.99-git
+```
+
+Doing this works great and has the advantage of being easy to achieve on
+your favorite CI platform.
+
+That said, you’ll really get the most of it by using
+[Cuirass](https://guix.gnu.org/en/cuirass), a CI tool designed for and
+tightly integrated with Guix. Using it is more work than using a hosted
+CI tool because you first need to set it up, but that setup phase is
+greatly simplified if you use its Guix System
+[service](https://guix.gnu.org/manual/devel/en/html_node/Continuous-Integration.html).
+Going back to our example, we give Cuirass a spec file that goes like
+this:
+
+```scheme
+;; Cuirass spec file to build all the packages of the ‘guile’ channel.
+(list (specification
+ (name "guile")
+ (build '(channels guile))
+ (channels
+ (append (list (channel
+ (name 'guile)
+ (url "https://git.savannah.gnu.org/git/guile.git";)
+ (branch "main")))
+ %default-channels))))
+```
+
+It differs from what you’d do with other CI tools in two important ways:
+
+ - Cuirass knows it’s tracking *two* channels, `guile` and `guix`.
+ Indeed, our own `guile` package depends on many packages provided by
+ the `guix` channel—GCC, the GNU libc, libffi, and so on. Changes to
+ packages from the `guix` channel can potentially influence our
+ `guile` build and this is something we’d like to see as soon as
+ possible as Guile developers.
+ - Build results are not thrown away: they can be distributed as
+
[*substitutes*](https://guix.gnu.org/manual/devel/en/html_node/Substitutes.html)
+ so that users of our `guile` channel transparently get pre-built
+ binaries!
+
+From a developer’s viewpoint, the end result is this [status
+page](https://ci.guix.gnu.org/jobset/guile) listing *evaluations*: each
+evaluation is a combination of commits of the `guix` and `guile`
+channels providing a number of *jobs*—one job per package defined in
+`guile-package.scm` times the number of target architectures.
+
+As for substitutes, they come for free! As an example, our `guile`
+jobset being built on ci.guix.gnu.org, one automatically gets
+substitutes for it from ci.guix.gnu.org. It’s as simple as this.
+
+# Bonus: Build manifest
+
+The Cuirass spec above is convenient: it builds every package in our
+channel, which includes a few variants. However, this might may be
+insufficiently expressive in some cases: one might want specific
+cross-compilation jobs, transformations, Docker images, RPM/Deb
+packages, or even system tests.
+
+To achieve that, you can write a
+[*manifest*](https://guix.gnu.org/manual/devel/en/html_node/Writing-Manifests.html).
+The one we have for Guile has entries for the package variants we
+defined above, as well as additional variants and cross builds:
+
+```scheme
+;; This is ‘.guix/manifest.scm’ FIXME: move modules to .guix/modules/.
+(use-modules (guix)
+ (guix profiles)
+ (guile-package)) ;import our own package module
+
+(define* (package->manifest-entry* package system
+ #:key target)
+ "Return a manifest entry for PACKAGE on SYSTEM, optionally cross-compiled to
+TARGET."
+ (manifest-entry
+ (inherit (package->manifest-entry package))
+ (name (string-append (package-name package) "." system
+ (if target
+ (string-append "." target)
+ "")))
+ (item (with-parameters ((%current-system system)
+ (%current-target-system target))
+ package))))
+
+(define native-builds
+ (manifest
+ (append (map (lambda (system)
+ (package->manifest-entry* guile system))
+
+ '("x86_64-linux" "i686-linux"
+ "aarch64-linux" "armhf-linux"
+ "powerpc64le-linux"))
+ (map (lambda (guile)
+ (package->manifest-entry* guile "x86_64-linux"))
+ (cons (package
+ (inherit (package-with-c-toolchain
+ guile
+ `(("clang-toolchain"
+ ,(specification->package
+ "clang-toolchain")))))
+ (name "guile-clang"))
+ (list guile-without-threads
+ guile-without-networking
+ guile-debug
+ guile-strict-typing))))))
+
+(define cross-builds
+ (manifest
+ (map (lambda (target)
+ (package->manifest-entry* guile "x86_64-linux"
+ #:target target))
+ '("i586-pc-gnu"
+ "aarch64-linux-gnu"
+ "riscv64-linux-gnu"
+ "i686-w64-mingw32"
+ "x86_64-linux-gnu"))))
+
+(concatenate-manifests (list native-builds cross-builds))
+```
+
+We won’t go into the details of this manifest; suffice to say that it
+provides additional flexibility. We now need to tell Cuirass to build
+this manifest, which is done with a spec slightly different from the
+previous one:
+
+```scheme
+;; Cuirass spec file to build all the packages of the ‘guile’ channel.
+(list (specification
+ (name "guile")
+ (build '(manifest ".guix/manifest.scm"))
+ (channels
+ (append (list (channel
+ (name 'guile)
+ (url "https://git.savannah.gnu.org/git/guile.git";)
+ (branch "main")))
+ %default-channels))))
+```
+
+This is it!
+
+# Wrapping up
+
+We picked Guile as the running example in this post and you can see the
+result here:
+
+ -
[`.guix-channel`](https://git.savannah.gnu.org/cgit/guile.git/tree/.guix-channel?id=36fd2b4920ae926c79b936c29e739e71a6dff2bc);
+ -
[`.guix/modules/guile-package.scm`](https://git.savannah.gnu.org/cgit/guile.git/tree/build-aux/guix/guile-package.scm?id=36fd2b4920ae926c79b936c29e739e71a6dff2bc)
+ with the top-level `guix.scm` symlink;
+ -
[`.guix/manifest.scm`](https://git.savannah.gnu.org/cgit/guile.git/tree/build-aux/manifest.scm?id=36fd2b4920ae926c79b936c29e739e71a6dff2bc).
+
+These days, repositories are commonly splattered with dot files for
+various tools: `.envrc`, `.gitlab-ci.yml`, `.github/workflows`,
+`Dockerfile`, `.buildpacks`, `Aptfile`, `requirements.txt`, and whatnot.
+It may sound like we’re proposing a bunch of *additional* files, but in
+fact those files are expressive enough to *supersede* most or all of
+those listed above.
+
+With a couple of files, we get support for:
+
+ - development environments (`guix shell`);
+ - pristine test builds, including for package variants and for
+ cross-compilation (`guix build`);
+ - continuous integration;
+ - continuous delivery to users (*via* the channel and with pre-built
+ binaries);
+ - generation of derivative build artifacts such as Docker images or
+ Deb/RPM packages (`guix pack`).
+
+At the Guix headquarters, we’re quite happy about the result. We’ve
+been building a unified tool set for reproducible software deployment
+when all the rage is on specialized tools building on one another;
+hopefully this is an illustration of how you as a developer can benefit
+from it!
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- branch master updated: website: Add draft about packages as channels.,
Ludovic Courtès <=