[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#61765] custom toolchain blog post
|
From: |
Mitchell Schmeisser |
|
Subject: |
[bug#61765] custom toolchain blog post |
|
Date: |
Fri, 24 Feb 2023 13:51:56 -0500 |
Here is a rough draft of my toolchain post.
I hope it can make an interesting contribution to the Guix literature.
- Mitchell
>From 4f6c43091ffd67cdbc5f041e496f61bc8a06070e Mon Sep 17 00:00:00 2001
From: Mitchell Schmeisser <mitchellschmeisser@librem.one>
Date: Fri, 24 Feb 2023 13:02:05 -0500
Subject: [PATCH] website: Add custom toolchain blog post
* website/posts/custom-toolchains-with-guix.md: New file.
---
website/posts/custom-toolchains-with-guix.md | 557 +++++++++++++++++++
1 file changed, 557 insertions(+)
create mode 100644 website/posts/custom-toolchains-with-guix.md
diff --git a/website/posts/custom-toolchains-with-guix.md
b/website/posts/custom-toolchains-with-guix.md
new file mode 100644
index 0000000..f73f1ab
--- /dev/null
+++ b/website/posts/custom-toolchains-with-guix.md
@@ -0,0 +1,557 @@
+# Table of Contents
+
+1. [Overview](#org2633a51)
+2. [Anatomy of a toolchain](#orgc440e9e)
+3. [Bootstrapping a Toolchain](#orgd42b6c3)
+4. [Defining the Packages](#org55042c5)
+ 1. [Binutils](#org67da1ec)
+ 2. [GCC sans libc](#org82d6f83)
+ 3. [Newlib(-nano)](#orgf6bafbc)
+ 4. [Complete toolchain](#org052f2a2)
+5. [Integrating with Zephyr Build System](#orgc3f87f4)
+ 1. [Testing](#org9f3c314)
+
+All code is available at
[guix-zephyr](https://github.com/paperclip4465/guix-zephyr) channel.
+
+
+<a id="org2633a51"></a>
+
+# Overview
+
+In order to deploy embedded software using Guix we first need to teach Guix
+how to build it. Since Guix bootstraps everything this means we must teach Guix
+how to build our toolchain.
+
+The [Zephyr Project](https://zephyrproject.org) uses its own fork of GCC with
custom configs for
+the architectures supported by the project.
+
+
+<a id="orgc440e9e"></a>
+
+# Anatomy of a toolchain
+
+Toolchains are responsible for taking high level descriptions of programs
+and lowering them down to a series of equivalent machine instructions.
+This process involves more than just a compiler. The compiler uses the
`binutils`
+to manipulate it's internal representation down to a given architecture.
+It also needs the use of the C standard library as well as a few other
libraries
+needed for some compiler optimizations.
+
+The C library provides the interface to the underlying kernel. System calls
like `write`
+and `read` are provided by `Glibc` on most Linux distributions.
+
+In embedded systems smaller implementations like `newlib` and `newlib-nano`
are used.
+
+
+<a id="orgd42b6c3"></a>
+
+# Bootstrapping a Toolchain
+
+In order to compile GCC we need a C library that's been compiled for
+our target architecture. How can we cross compile our C library if we
+need our C library to build a cross compiler? The solution is to build
+a simpler compiler that doesn't require the C library to function.
+It will not be capable of as many optimizations and it will be very slow,
+however it will be able to build the C libraries as well as the complete
version
+of GCC.
+
+In order to build the simpler compiler we need to compile the `binutils` to
+work with our target architecture.
+The `binutils` can be bootstrapped with our host GCC and have no target
dependencies.
+
+[For more information read
this.](https://crosstool-ng.github.io/docs/toolchain-construction/)
+
+Doesn't sound so bad right? It isn't… in theory.
+However internet forums since time immemorial have been
+littered with the laments of those who came before.
+From incorrect versions of `ISL` to the wrong C library being linked
+or the host linker being used, etc.
+The one commonality between all of these issues is the environment.
+Building a cross toolchain is difficult because isolating build
+environments is difficult.
+
+In fact as of `v0.14.2` the zephyr SDK repository took down the build
+instructions and posted a sign that read
+"Building this is too complicated, don't worry about it."<sup><a id="fnr.1"
class="footref" href="#fn.1" role="doc-backlink">1</a></sup>
+
+We will neatly side step all of these problems and not
+risk destroying or polluting our host system with garbage
+by using Guix to manage our environments for us.
+
+Our toolchain only requires the first pass compiler because
+newlib(-nano) is statically linked and introduced to the toolchain
+by normal package composition.
+
+
+<a id="org55042c5"></a>
+
+# Defining the Packages
+
+All of the base packages are defined in `zephyr/packages/zephyr.scm`.
+Zephyr modules are defined in `zephyr/packages/zephyr-xyz.scm`, following
+the pattern of other module systems implemented by Guix.
+
+
+<a id="org67da1ec"></a>
+
+## Binutils
+
+First thing we need to build is the `arm-zephyr-eabi` binutils.
+This is very easy in Guix.
+
+ (define-module (zephyr packages zephyr)
+ #:use-module (guix packages))
+
+ (define-public arm-zephyr-eabi-binutils
+ (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
+ (package
+ (inherit xbinutils)
+ (name "arm-zephyr-eabi-binutils")
+ (version "2.38")
+ (source
+ (origin (method git-fetch)
+ (uri (git-reference
+ (url
"https://github.com/zephyrproject-rtos/binutils-gdb";)
+ (commit
"6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
+ (file-name (git-file-name name version))
+ (sha256 (base32
"0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
+ (arguments
+ `(#:tests? #f
+ ,@(substitute-keyword-arguments (package-arguments
xbinutils)
+ ((#:configure-flags flags)
+ `(cons "--program-prefix=arm-zephyr-eabi-"
,flags)))))
+ (native-inputs
+ (append
+ (list texinfo
+ bison
+ flex
+ gmp
+ dejagnu)
+ (package-native-inputs xbinutils)))
+ (home-page "https://zephyrproject.org";)
+ (synopsis "binutils for zephyr RTOS"))))
+
+The function `cross-binutils` returns a package which has been
+configured for the given gnu triplet. We simply inherit that package
+and replace the source.
+The zephyr build system expects the binutils to be prefixed with
+`arm-zephyr-eabi-` which is accomplished by adding another flag to the
+`#:configure-flags` argument.
+
+We can test our package definition using the `-L` flag with `guix build`
+to add our packages.
+
+ guix build -L guix-zephyr zephyr-binutils
+
+ /gnu/store/...-zephyr-binutils-2.38
+
+This directory contains the results of `make install`.
+
+
+<a id="org82d6f83"></a>
+
+## GCC sans libc
+
+This one is a bit more involved. Don't be afraid!
+This version of GCC wants ISL version 0.15. It's easy enough
+to make that happen. Inherit the current version of ISL and swap
+out the source and update the version. For most packages the build process
doesn't
+change that much between versions.
+
+ (define-public isl-0.15
+ (package
+ (inherit isl)
+ (version "0.15")
+ (source (origin
+ (method url-fetch)
+ (uri (list (string-append
"mirror://sourceforge/libisl/isl-"
+
version ".tar.gz")))
+ (sha256
+ (base32
+
"11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))
+
+Like the binutils, there is a function for creating cross-gcc packages.
+This one accepts keywords specifying which binutils and libc to use.
+If libc isn't given (like here), gcc is configured with many options disabled
+to facilitate being built without libc. Therefore we need to add the extra
options
+we want <sup><a id="fnr.2" class="footref" href="#fn.2"
role="doc-backlink">2</a></sup>
+
+ (define-public gcc-arm-zephyr-eabi-12
+ (let ((xgcc (cross-gcc "arm-zephyr-eabi"
+ #:xbinutils
zephyr-binutils)))
+ (package
+ (inherit xgcc)
+ (version "12.1.0")
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url
"https://github.com/zephyrproject-rtos/gcc";)
+ (commit
"0218469df050c33479a1d5be3e5239ac0eb351bf")))
+ (file-name (git-file-name
(package-name xgcc) version))
+ (sha256
+ (base32
"1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
+ (patches
+ (search-patches
"gcc-12-cross-environment-variables.patch"
+
"gcc-cross-gxx-include-dir.patch"))))
+ (native-inputs
+ (modify-inputs (package-native-inputs xgcc)
+ ;; Get rid of stock ISL
+ (delete "isl")
+ ;; Add additional dependencies that xgcc doesn't have
+ ;; including our special ISL
+ (prepend flex
+ perl
+ python-3
+ gmp
+ isl-0.15
+ texinfo
+ python
+ mpc
+ mpfr
+ zlib)))
+ (arguments
+ (substitute-keyword-arguments (package-arguments xgcc)
+ ((#:phases phases)
+ `(modify-phases ,phases
+ (add-after 'unpack 'fix-genmultilib
+ (lambda _
+ (substitute* "gcc/genmultilib"
+ (("#!/bin/sh") (string-append "#!"
(which "sh"))))
+ #t))
+
+ (add-after 'set-paths
'augment-CPLUS_INCLUDE_PATH
+ (lambda* (#:key inputs #:allow-other-keys)
+ (let ((gcc (assoc-ref inputs "gcc")))
+ ;; Remove the default compiler from
CPLUS_INCLUDE_PATH to
+ ;; prevent header conflict with the
GCC from native-inputs.
+ (setenv "CPLUS_INCLUDE_PATH"
+ (string-join
+ (delete
(string-append gcc "/include/c++")
+
(string-split (getenv "CPLUS_INCLUDE_PATH")
+
#\:))
+ ":"))
+ (format #t
+ "environment
variable `CPLUS_INCLUDE_PATH' changed to ~a~%"
+ (getenv
"CPLUS_INCLUDE_PATH"))
+ #t)))))
+
+ ((#:configure-flags flags)
+ ;; The configure flags are largely identical to the
flags used by the
+ ;; "GCC ARM embedded" project.
+ `(append (list "--enable-multilib"
+ "--with-newlib"
+
"--with-multilib-list=rmprofile"
+
"--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
+ "--enable-plugins"
+
"--disable-decimal-float"
+ "--disable-libffi"
+ "--disable-libgomp"
+ "--disable-libmudflap"
+ "--disable-libquadmath"
+ "--disable-libssp"
+
"--disable-libstdcxx-pch"
+ "--disable-nls"
+ "--disable-shared"
+ "--disable-threads"
+ "--disable-tls"
+ "--with-gnu-ld"
+ "--with-gnu-as"
+
"--enable-initfini-array")
+ (delete "--disable-multilib"
,flags)))))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "CROSS_C_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include")))
+ (search-path-specification
+ (variable "CROSS_CPLUS_INCLUDE_PATH")
+ (files '("arm-zephyr-eabi/include"
+ "arm-zephyr-eabi/c++"
+
"arm-zephyr-eabi/c++/arm-zephyr-eabi")))
+ (search-path-specification
+ (variable "CROSS_LIBRARY_PATH")
+ (files '("arm-zephyr-eabi/lib")))))
+ (home-page "https://zephyrproject.org";)
+ (synopsis "GCC for zephyr RTOS"))))
+
+This GCC can be built like so.
+
+ guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi
+
+ /gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
+ /gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0
+
+Great! We now have our stage-1 compiler.
+
+
+<a id="orgf6bafbc"></a>
+
+## Newlib(-nano)
+
+The newlib package package is quite straight forward (relatively).
+It is mostly adding in the relevent configuration flags and patching
+the files the `patch-shebangs` phase missed.
+
+ (define-public zephyr-newlib
+ (package
+ (name "zephyr-newlib")
+ (version "3.3")
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url
"https://github.com/zephyrproject-rtos/newlib-cygwin";)
+ (commit
"4e150303bcc1e44f4d90f3489a4417433980d5ff")))
+ (sha256
+ (base32
"08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:out-of-source? #t
+ #:configure-flags '("--target=arm-zephyr-eabi"
+
"--enable-newlib-io-long-long"
+
"--enable-newlib-io-float"
+
"--enable-newlib-io-c99-formats"
+
"--enable-newlib-retargetable-locking"
+
"--enable-newlib-lite-exit"
+
"--enable-newlib-multithread"
+
"--enable-newlib-register-fini"
+
"--enable-newlib-extra-sections"
+
"--disable-newlib-wide-orient"
+
"--disable-newlib-fseek-optimization"
+
"--disable-newlib-supplied-syscalls"
+
"--disable-newlib-target-optspace"
+ "--disable-nls")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'fix-references-to-/bin/sh
+ (lambda _
+ (substitute*
'("libgloss/arm/cpu-init/Makefile.in"
+
"libgloss/arm/Makefile.in"
+
"libgloss/libnosys/Makefile.in"
+
"libgloss/Makefile.in")
+ (("/bin/sh") (which "sh")))
+ #t)))))
+ (native-inputs
+ `(("xbinutils" ,zephyr-binutils)
+ ("xgcc" ,gcc-arm-zephyr-eabi-12)
+ ("texinfo" ,texinfo)))
+ (home-page "https://www.sourceware.org/newlib/";)
+ (synopsis "C library for use on embedded systems")
+ (description "Newlib is a C library intended for use on embedded
+ systems. It is a conglomeration of several library parts that are
easily
+ usable on embedded products.")
+ (license (license:non-copyleft
+
"https://www.sourceware.org/newlib/COPYING.NEWLIB";))))
+
+And the build.
+
+ guix build -L guix-zephyr zephyr-newlib
+
+ /gnu/store/...-zephyr-newlib-3.3
+
+
+<a id="org052f2a2"></a>
+
+## Complete toolchain<sup><a id="fnr.3" class="footref" href="#fn.3"
role="doc-backlink">3</a></sup>
+
+Now that we've got the individual tools it's time to create our complete
toolchain.
+For this we need to do some package transformations.
+Because these transformations are going to have to be done for every
combination of
+gcc/newlib it is best to create a function which we can reuse for every version
+of the SDK.
+
+ (define (arm-zephyr-eabi-toolchain xgcc newlib version)
+ "Produce a cross-compiler zephyr toolchain package with the compiler
XGCC and the C
+ library variant NEWLIB."
+ (let ((newlib-with-xgcc (package (inherit newlib)
+
(native-inputs
+
(alist-replace "xgcc" (list xgcc)
+
(package-native-inputs newlib))))))
+ (package
+ (name (string-append "arm-zephyr-eabi"
+ (if (string=?
(package-name newlib-with-xgcc)
+
"newlib-nano")
+ "-nano" "")
+ "-toolchain"))
+ (version version)
+ (source #f)
+ (build-system trivial-build-system)
+ (arguments
+ '(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (ice-9 match)
+ (guix build union)
+ (guix build utils))
+ (let ((out (assoc-ref %outputs "out")))
+ (mkdir-p out)
+ (match %build-inputs
+ (((names . directories) ...)
+ (union-build (string-append out
"/arm-zephyr-eabi")
+ directories)
+ #t))))))
+ (inputs
+ `(("binutils" ,zephyr-binutils)
+ ("gcc" ,xgcc)
+ ("newlib" ,newlib-with-xgcc)))
+ (synopsis "Complete GCC tool chain for ARM zephyrRTOS
development")
+ (description "This package provides a complete GCC tool chain
for ARM
+ bare metal development with zephyr rtos. This includes the GCC
arm-zephyr-eabi cross compiler
+ and newlib (or newlib-nano) as the C library. The supported programming
+ language is C.")
+ (home-page (package-home-page xgcc))
+ (license (package-license xgcc)))))
+
+This function creates a special package which consists of the toolchain in a
special directory hierarchy, i.e `arm-zephyr-eabi/`.
+Our complete toolchain definition looks like this.
+
+ (define-public arm-zephyr-eabi-toolchain-0.15.0
+ (arm-zephyr-eabi-toolchain
+ gcc-arm-zephyr-eabi-12
+ zephyr-newlib
+ "0.15.0"))
+
+To build:
+
+ guix build -L guix-zephyr arm-zephyr-eabi-toolchain
+
+ /gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0
+
+
+<a id="orgc3f87f4"></a>
+
+# Integrating with Zephyr Build System
+
+Zephyr uses CMake as it's build system. It contains numerous CMake
+files in both the so-called `ZEPHYR_BASE`, the zephyr source code
+repository, as well as a handful in the SDK
+which help select the correct toolchain for a given board.
+
+There are standard locations the build system will look for the
+SDK. We are not using any of them. Our SDK lives in the store,
+immutable forever. According to
[this](https://docs.zephyrproject.org/latest/develop/west/without-west.html)
the variable
+`ZEPHYR_SDK_INSTALL_DIR` needs to point to our custom spot.
+
+We also need to grab the cmake files from the
[repository](https://github.com/zephyrproject-rtos/sdk-ng) and create a
+file `sdk_version` which contains the version string `ZEPHYR_BASE`
+uses to find a compatible SDK.
+
+Along with the SDK proper we need to include a number of python
+packages required by the build system<sup><a id="fnr.4" class="footref"
href="#fn.4" role="doc-backlink">4</a></sup>.
+
+ (define-public zephyr-sdk
+ (package
+ (name "zephyr-sdk")
+ (version "0.15.0")
+ (home-page "https://zephyrproject.org";)
+ (source (origin (method git-fetch)
+ (uri (git-reference
+ (url
"https://github.com/zephyrproject-rtos/sdk-ng";)
+ (commit "v0.15.0")))
+ (file-name (git-file-name name
version))
+ (sha256 (base32
"04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
+ (build-system trivial-build-system)
+ (arguments
+ `(#:modules ((guix build union)
+ (guix build utils))
+ #:builder
+ (begin
+ (use-modules (guix build union)
+ (ice-9 match)
+ (guix build utils))
+ (let* ((out (assoc-ref %outputs "out"))
+ (cmake-scripts (string-append
(assoc-ref %build-inputs "source")
+
"/cmake"))
+ (sdk-out (string-append out
"/zephyr-sdk-0.15.0")))
+ (mkdir-p out)
+
+ (match (assoc-remove! %build-inputs "source")
+ (((names . directories) ...)
+ (union-build sdk-out directories)))
+
+ (copy-recursively cmake-scripts
+ (string-append
sdk-out "/cmake"))
+
+ (with-directory-excursion sdk-out
+ (call-with-output-file "sdk_version"
+ (lambda (p)
+ (format p "0.15.0")))
+ #t)))))
+ (propagated-inputs
+ (list
+ arm-zephyr-eabi-toolchain-0.15.0
+ zephyr-binutils
+ python-3
+ python-pyelftools
+ python-pykwalify
+ python-pyyaml
+ python-packaging
+ dtc))
+ (native-search-paths
+ (list (search-path-specification
+ (variable "ZEPHYR_SDK_INSTALL_DIR")
+ (files '("")))))
+ (synopsis "SDK for zephyrRTOS")
+ (description "zephyr-sdk contains bundles a complete gcc
toolchain as well
+ as host tools like dtc, openocd, qemu, and required python
packages.")
+ (license license:apsl2)))
+
+
+<a id="org9f3c314"></a>
+
+## Testing
+
+In order to test we will need an environment with the SDK installed.
+We can take advantage of `guix shell` to avoid installing test
+packages into our home environment. This way if it causes problems we
+can just exit the shell and try again.
+
+ guix shell -L guix-zephyr zephyr-sdk cmake ninja git
+
+`ZEPHYR_BASE` can be cloned into a temporary workspace to test our
+toolchain functionality.<sup><a id="fnr.5" class="footref" href="#fn.5"
role="doc-backlink">5</a></sup>
+
+ mkdir /tmp/zephyr-project
+ cd /tmp/zephyr-project
+ git clone https://github.com/zephyrproject-rtos/zephyr
+ export ZEPHYR_BASE=/tmp/zephyr-project/zephyr
+
+In order to build for the test board (k64f in this case) we need to
+get a hold of the vendor Hardware Abstraction Layers and CMSIS.<sup><a
id="fnr.6" class="footref" href="#fn.6" role="doc-backlink">6</a></sup>
+
+ git clone https://github.com/zephyrproject-rtos/hal_nxp &&
+ git clone https://github.com/zephyrproject-rtos/cmsis
+
+To inform the build system about this module we pass it in with
+`-DZEPHYR_MODULES=` which is a semicolon separated list of paths
+containing a module.yml file.
+
+To build the hello world sample we use the following incantation.
+
+ cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
+ -GNinja \
+ -DBOARD=frdm_k64f \
+ -DBUILD_VERSION=3.1.0 \
+
-DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
+ && ninja -Cbuild
+
+If everything is set up correctly we will end up with a `./build`
+directory with all our build artifacts. The SDK is installed correctly!
+
+
+# Footnotes
+
+<sup><a id="fn.1" href="#fnr.1">1</a></sup> I'm paraphrasing, but [not by
much](https://github.com/zephyrproject-rtos/sdk-ng/tree/v0.14.2#build-process).
+
+<sup><a id="fn.2" href="#fnr.2">2</a></sup> I got them from the SDK
configuration scripts on the [sdk
github](https://github.com/zephyrproject-rtos/sdk-ng) as well as the
+commits to use for each of the tools.
+
+<sup><a id="fn.3" href="#fnr.3">3</a></sup> *Mostly* complete. libstdc++ does
not build because
+\`arm-zephyr-eabi\` is not \`arm-none-eabi\` so a dynamic link check is
+performed/failed. I cannot figure out how crosstool-ng handles this.
+
+<sup><a id="fn.4" href="#fnr.4">4</a></sup> This is no longer the case. Now
python packages are provided by the zephyr package
+which was not available when this was being developed.
+
+<sup><a id="fn.5" href="#fnr.5">5</a></sup> For now. Eventually we will need
to create a package for `zephyr-base` that
+our guix zephyr-build-system can use.
+
+<sup><a id="fn.6" href="#fnr.6">6</a></sup> These will also need to become
guix packages to allow the build system to compose modules.
--
2.39.1
- [bug#61765] custom toolchain blog post,
Mitchell Schmeisser <=