Re: The Shepherd on Fibers

[pelzflorian (Florian Pelz)]

Hi Ludo!  That you add fibers to Shepherd is great news.

On Wed, Mar 23, 2022 at 11:36:30PM +0100, Ludovic Courtès wrote:
> Fibers is used in a single-threaded fashion, which is the main
> constraint for shepherd since it forks.  That also means that fibers
> cannot be preempted, so it’s fully cooperative scheduling.

To understand what you mean, I needed to read shepherd commit
934204fbd158aa9fbd42914b89aa960f99eef125

+      ;; Run Fibers in such a way that it does not create any POSIX thread,
+      ;; because POSIX threads and 'fork' cannot be used together

So shepherd service authors still cannot write blocking code but need
to yield?


> I’ve done some Guix System testing in VMs and didn’t notice any major
> issues.  I’d like to merge that branch in ‘master’ and to eventually
> release it as 0.9.0 (with or without socket activation, we’ll see.)
> Hopefully, we could be running it within a couple of weeks.

I wanted to test too how much time it takes to boot to GDM on my slow
2010 dual-core Macbook, but I can’t quite figure out each shepherd-
service failing to build.  How did you test the new shepherd in Guix?

florian@florianmacbook ~/git/guix [env]$ gzip -cd 
/var/log/guix/drvs/f3/g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv.gz
 
Backtrace:
          16 (primitive-load "/gnu/store/4qk3x8dl8hxnwry637kq4ahh2z2?")
In ice-9/eval.scm:
    619:8 15 (_ #(#<directory (guile-user) 7ffff6fdec80> #<module ?>))
    159:9 14 (_ #(#<directory (guile-user) 7ffff6fdec80> #<module ?>))
In ice-9/boot-9.scm:
  3326:17 13 (resolve-interface (shepherd service) #:select _ #:hide ?)
In ice-9/threads.scm:
    390:8 12 (_ _)
In ice-9/boot-9.scm:
  3252:13 11 (_)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3536:20  9 (_)
   2835:4  8 (save-module-excursion #<procedure 7ffff5f4ea50 at ice-?>)
  3556:26  7 (_)
In unknown file:
           6 (primitive-load-path "shepherd/service" #<procedure 7ff?>)
In shepherd/service.scm:
     26:0  5 (_)
In ice-9/boot-9.scm:
   3409:4  4 (define-module* _ #:filename _ #:pure _ #:version _ # _ ?)
  2594:24  3 (call-with-deferred-observers #<procedure 7ffff5f3b410 ?>)
  3422:24  2 (_)
   222:17  1 (map1 (((fibers)) ((fibers scheduler) #:select (#)) # ?))
   3329:6  0 (resolve-interface (fibers) #:select _ #:hide _ #:prefix ?)

ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (fibers)



Anyway, testing my boot time is not so important.  I’d probably have
to change something in gnu/services/shepherd.scm.  Or I should rebuild
shepherd in a less crude way.

Feel free to ignore me, though for completeness, here’s what I did:

* Pull shepherd wip-fibers commit
  abbdd1e8b0f8bd4b3952efc5614f560aab9a79bf.

* I changed its Makefile.am and added -O1 to '$(GUILD) compile'
  because Guix normally does likewise in an origin snippet.

* Then `guix shell -D shepherd autoconf automake guile-fibers help2man
  texinfo -- sh -c 'autoreconf -vif && ./configure
  --prefix=/home/florian/git/shepherd/out && make && make install'`.

* Then to Guix I added to gnu/packages/admin.scm:

(define-public new-shepherd
  (package
    (name "shepherd")
    (version "0.9.0")
    (source (local-file "/home/florian/git/shepherd/out" #:recursive? #t))
    (build-system copy-build-system)
    (propagated-inputs (list guile-fibers))
    (arguments
       `(#:install-plan '(("./" ""))))
    …)) ;; and synopsis, description, license, homepage

and at the top #:use-module (gnu packages guile-xyz) to add
guile-fibers to new-shepherd’s propagated inputs.  Also #:use-module
(guix build-system copy).

* Lastly I grafted

(define-public shepherd
  (package
    (name "shepherd")
    (replacement new-shepherd)
    …

* Compiled Guix and reconfigured.  Building
  /gnu/store/f3g9drzlg8vkzp6z81dcdsnfnaqa2anx-shepherd-avahi-daemon.go.drv
  failed.

The same error happens when using new-shepherd as shepherd and
rebuilding the GNOME world.

Regards,
Florian