Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi Maxim!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

>> The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.
>
> Very exciting work!  I can't wait to have my 5 min boot reduced (mostly
> busy-waiting on the child-hurd VM to start) ;-).

Same here!  The childhurd service will need some help as it’s currently
written in a way that would block.

> Interesting; in a (unmerged) patch I had made to mcron, I had extended
> the select loop in a way that it'd detect available output and use a
> suspendable port to accomplish this; perhaps Fibers would be a
> cleaner/more straightforward way to accomplish this.  Also, I was
> particularly happy about how flexible the timestamp/metadata prefixed to
> each line could be configured by a user; you may want to steal ideas
> from it! [0]
>
> [0]  https://lists.gnu.org/archive/html/bug-mcron/2021-08/msg00008.html

Oh nice!  I hope that patch will eventually make it into mcron (or we
could use our own version!).

For the timestamp format there’s currently a parameter, which is not
exposed, and defaults to syslog style.

> I see.  I was puzzle by this configure.ac check:
>
> +dnl Check for extra dependencies.
> +GUILE_MODULE_AVAILABLE([have_fibers], [(fibers)])
> +if test "x$have_fibers" != "xyes"; then
> +  AC_MSG_ERROR([Fibers is missing; please install it.])
> +fi
> +
> +dnl Make sure Fibers does not create POSIX threads: since shepherd
> +dnl forks, it must be single-threaded.
> +AC_CACHE_CHECK([whether Fibers might create POSIX threads],
> +  [ac_cv_fibers_creates_pthreads],
> +  [GUILE_CHECK([retval],
> +    [(use-modules (fibers))
> +     (set! (@ (ice-9 threads) call-with-new-thread)
> +       (lambda _ (throw 'new-thread!)))
> +     (run-fibers (lambda () (spawn-fiber (lambda () 1)))
> +                 #:parallelism 1 #:hz 0)])
> +   if test "$retval" = 0; then
> +     ac_cv_fibers_creates_pthreads="no"
> +   else
> +     ac_cv_fibers_creates_pthreads="yes"
> +   fi])
> +if test "x$ac_cv_fibers_creates_pthreads" = "xyes"; then
> +  AC_MSG_ERROR([Fibers creates POSIX threads behind our back; aborting.])
> +fi
>
>
> In which scenario would Fibers create unwanted POSIX threads?

It shouldn’t happen with #:parallelism 1 #:hz 0.  Here I’m just being
super defensive: it’s PID 1 after all, and if future Fibers versions
break that assumption for some reason, I’d rather detect it early.

> 1. In fae59fb * build: Capture the source and object directories of
> Fibers.
>
> Why must we even capture the Fibers source/compiled objects directories?

We don’t have to, but if we don’t do it here, then the ‘shepherd’
package definition will have to use ‘wrap-program’ to do it for us.
Since there’s just one dependency, I thought we might as well do it
upstream.

> 2. nitpick; in "service: 'read-pid-file' no longer blocks":
>
> -;; Copyright (C) 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 
> Ludovic Courtès <ludo@gnu.org>
> +;; Copyright (C) 2013-2022 Ludovic Courtès <ludo@gnu.org>
>
> While I agree this reads better, I've recently stumbled in info
> '(maintain) Copyright Notices' on:
>
>       You can use a range (‘2008-2010’) instead of listing individual years
>    (‘2008, 2009, 2010’) if and only if: 1) every year in the range,
>    inclusive, really is a “copyrightable” year that would be listed
>    individually; _and_ 2) you make an explicit statement in a ‘README’ file
>    about this usage.
>
> So you'd want to add an explicit statement in the README about it.

Yes, will do (it’s typical FSF pedantry; I remember the time it took
before this anecdotal-looking paragraph would end up in that document,
as if software freedom was at stake :-)).

> 3. I was a bit skeptical about the ability to configure the encoding of
> the log file; in which situation would switching it to something else
> than UTF-8 be useful?

I agree; the log files produced by shepherd are always UTF-8.

What’s configurable is the encoding of the data produced by daemons.
It’s probably rarely useful, but if you have a daemon that writes, say,
ISO-8859-1 strings, you can say so.  That way, the output of that daemon
is properly decoded as ISO-8859-1; the log file produced by shepherd
remains UTF-8 no matter what.

Thanks for taking a look!

Since that initial message I added support for inetd-style service
startup.  Among the services I use, it could be use for sshd (OpenSSH),
bitlbee, and dicod.  More on that later…

Ludo’.