branch master updated: website: Add post about using glibc-hwcaps.

[Efraim Flashner]

This is an automated email from the git hooks/post-receive script.

efraim pushed a commit to branch master
in repository guix-artwork.

The following commit(s) were added to refs/heads/master by this push:
     new 787bae9  website: Add post about using glibc-hwcaps.
787bae9 is described below

commit 787bae9f8ef3b9db98dad60f50511e0b7d3fc46e
Author: Efraim Flashner <efraim@flashner.co.il>
AuthorDate: Sun Jan 14 10:17:54 2024 +0200

    website: Add post about using glibc-hwcaps.
    
    * website/posts/guix-packages-with-glibc-hwcaps.md: New file.
---
 website/posts/guix-packages-with-glibc-hwcaps.md | 183 +++++++++++++++++++++++
 1 file changed, 183 insertions(+)

diff --git a/website/posts/guix-packages-with-glibc-hwcaps.md 
b/website/posts/guix-packages-with-glibc-hwcaps.md
new file mode 100644
index 0000000..1d9cc16
--- /dev/null
+++ b/website/posts/guix-packages-with-glibc-hwcaps.md
@@ -0,0 +1,183 @@
+title: Building packages targeting psABIs
+author: Efraim Flashner
+tags: Packaging
+date: 2024-01-14 12:00:00
+---
+
+Starting with version 2.33, the GNU C library (glibc) grew the capability to
+search for shared libraries using additional paths, based on the hardware
+capabilities of the machine running the code.  This was a great boon for
+x86_64, which was first released in 2003, and has seen many changes in the
+capabilities of the hardware since then.  While it is extremely common for
+Linux distributions to compile for a baseline which encompasses all of an
+architecture, there is performance being left on the table by targeting such an
+old specification and not one of the newer revisions.
+
+One option used internally in glibc and in some other performance-critical
+libraries is [indirect functions, or
+IFUNCs](https://sourceware.org/glibc/wiki/GNU_IFUNC) (see also
+[here](https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/))
+The loader, `ld.so` uses them to pick function implementations optimized for
+the available CPU at load time.  GCC's (functional multi-versioning
+(FMV))[https://gcc.gnu.org/wiki/FunctionMultiVersioning] generates several
+optimized versions of functions, using the IFUNC mechanism so the approprate
+one is selected at load time. These are strategies which most
+performance-sensitive libraries do, but not all of them.
+
+With the `--tune` using [package
+transformation](https://guix.gnu.org/en/manual/devel/en/html_node/Package-Transformation-Options.html)
+option, Guix implements so-called [package
+multi-versioning](https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/),
+which creates package variants using compiler flags set to use optimizations
+targeted for a specific CPU.
+
+Finally - and we're getting to the central topic of this post! - glibc since
+version 2.33 supports another approach: `ld.so` would search not just the
+`/lib` folder, but also the `glibc-hwcaps` folders, which for x86_64 included
+`/lib/glibc-hwcaps/x86-64-v2`, `/lib/glibc-hwcaps/x86-64-v3` and
+`/lib/glibc-hwcaps/x86-64-v4`, corresponding to the psABI micro-architectures
+of the x86_64 architecture.  This means that if a library was compiled against
+the baseline of the architecture then it should be installed in `/lib`, but if
+it were compiled a second time, this time using (depending on the build
+instructions) `-march=x86-64-v2`, then the libraries could be installed in
+`/lib/glibc-hwcaps/x86-64-v2` and then glibc, using `ld.so`, would choose the
+correct library at runtime.
+
+These micro-architectures aren't a perfect match for the different hardware
+available, it is often the case that a particular CPU would satisfy the
+requirements of one tier and part of the next but would therefore only be able
+to use the optimizations provided by the first tier and not by the added
+features that the CPU also supports.
+
+This of course shouldn't be a problem in Guix; it's possible, and even
+encouraged, to adjust packages to be more useful for one's needs.  The problem
+comes from the search paths: `ld.so` will only search for the `glibc-hwcaps`
+directory if it has already found the base library in the preceding `/lib`
+directory.  This isn't a problem for distributions following the File System
+Hierarchy (FHS), but for Guix we will need to ensure that all the different
+versions of the library will be in the same output.
+
+With a little bit of planning this turns out to not be as hard as it sounds.
+Lets take for example, the [GNU Scientific
+Library](https://www.gnu.org/software/gsl/), gsl, a math library which helps
+with all sorts of numerical analysis.  First we create a procedure to generate
+our 3 additional packages, corresponding to the psABIs that are searched for in
+the `glibc-hwcaps` directory.
+
+``` scheme
+(define (gsl-hwabi psabi)
+  (package/inherit gsl
+    (name (string-append "gsl-" psabi))
+    (arguments
+     (substitute-keyword-arguments (package-arguments gsl)
+       ((#:make-flags flags #~'())
+        #~(append (list (string-append "CFLAGS=-march=" #$psabi)
+                        (string-append "CXXFLAGS=-march=" #$psabi))
+                  #$flags))
+       ((#:configure-flags flags #~'())
+        #~(append (list (string-append "--libdir=" #$output
+                                       "/lib/glibc-hwcaps/" #$psabi))
+                  #$flags))
+       ;; The building machine can't necessarily run the code produced.
+       ((#:tests? _ #t) #f)
+       ((#:phases phases #~%standard-phases)
+        #~(modify-phases #$phases
+            (add-after 'install 'remove-extra-files
+              (lambda _
+                (for-each (lambda (dir)
+                            (delete-file-recursively (string-append #$output 
dir)))
+                          (list (string-append "/lib/glibc-hwcaps/" #$psabi 
"/pkgconfig")
+                                "/bin" "/include" "/share"))))))))
+    (supported-systems '("x86_64-linux" "powerpc64le-linux"))
+    (properties `((hidden? . #t)
+                  (tunable? . #f)))))
+```
+
+We remove some directories and any binaries since we only want the libraries
+produced from the package; we want to use the headers and any other bits from
+the `main` package.  We then combine all of the pieces together to produce a
+package which can take advantage of the hardware on which it is run:
+
+``` scheme
+(define-public gsl-hwcaps
+  (package/inherit gsl
+    (name "gsl-hwcaps")
+    (arguments
+     (substitute-keyword-arguments (package-arguments gsl)
+       ((#:phases phases #~%standard-phases)
+        #~(modify-phases #$phases
+            (add-after 'install 'install-optimized-libraries
+              (lambda* (#:key inputs outputs #:allow-other-keys)
+                (let ((hwcaps "/lib/glibc-hwcaps/"))
+                  (for-each
+                    (lambda (psabi)
+                      (copy-recursively
+                        (string-append (assoc-ref inputs (string-append "gsl-" 
psabi))
+                                       hwcaps psabi)
+                        (string-append #$output hwcaps psabi))
+                  '("x86-64-v2" "x86-64-v3" "x86-64-v4"))))))))
+    (native-inputs
+     (modify-inputs (package-native-inputs gsl)
+                    (append (gsl-hwabi "x86-64-v2")
+                            (gsl-hwabi "x86-64-v3")
+                            (gsl-hwabi "x86-64-v4"))))
+    (supported-systems '("x86_64-linux"))
+    (properties `((tunable? . #f)))))
+```
+
+In this case the size of the final package is increased by about 13 MiB, from
+5.5 MiB to 18 MiB.  It is up to you if the speed-up from providing an optimized
+library is worth the size trade-off.
+
+To use this package as a replacement build input in a package
+`package-input-rewriting/spec` is a handy tool:
+
+``` scheme
+(define use-glibc-hwcaps
+ (package-input-rewriting/spec
+   ;; Replace some packages with ones built targeting custom packages build
+   ;; with glibc-hwcaps support.
+   `(("gsl" . ,(const gsl-hwcaps)))))
+
+(define-public inkscape-with-hwcaps
+  (package
+    (inherit (use-glibc-hwcaps inkscape))
+    (name "inkscape-with-hwcaps")))
+```
+
+Of the Guix supported architectures, x86_64-linux and powerpc64le-linux can
+both benefit from this new capability.
+
+Through the magic of newer versions of GCC and LLVM it is safe to use these
+libraries in place of the standard libraries while compiling packages; these
+compilers know about the `glibc-hwcap` directories and will purposefully link
+against the base library during build time, with glibc's `ld.so` choosing the
+optimized library at runtime.
+
+One possible use case for these libraries is crating [`guix
+pack`s](https://guix.gnu.org/en/manual/devel/en/html_node/Invoking-guix-pack.html)
+of packages to run on other systems.  By substituting these libraries it
+becomes possible to crate a `guix pack` which will have better performance than
+a standard package used in a `guix pack`.  This works even when the included
+libraries don't make use of the IFUNCs from glibc or functional
+multi-versioning from GCC.  Providing optimized yet portable pre-compiled
+binaries is a great way to take advantage of this feature.
+
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package manager
+and an advanced distribution of the GNU system that [respects user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the Hurd or the Linux
+kernel, or it can be used as a standalone operating system
+distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package
+management, per-user profiles, and garbage collection.  When used as a
+standalone GNU/Linux distribution, Guix offers a declarative,
+stateless approach to operating system configuration management.  Guix
+is highly customizable and hackable through
+[Guile](https://www.gnu.org/software/guile) programming interfaces and
+extensions to the [Scheme](http://schemers.org) language.