The Shepherd on Fibers

[Ludovic Courtès]

Hello Guix!

I have pushed a ‘wip-fibers’ branch of the Shepherd:

  https://git.savannah.gnu.org/cgit/shepherd.git/log/?h=wip-fibers

The goal is to make shepherd (the daemon) use Fibers¹ for concurrency.

Right now, actions in shepherd are all serialized: starting services,
waiting for their PID files, accepting client connections and serving
them (when running the ‘herd’ command), etc.  This slows down startup,
prevents things like running two clients at the same time, or serving a
client while waiting for a PID file.  In other words, it sucks.

Using Fibers, we can introduce concurrency with few changes to the code;
we can even do new things writing code that looks sequential.  This is
what the branch does.

To illustrate that, it introduces a new, incredible feature: logging!
As you know, shepherd had (euphemism ahead) limited support for logging,
essentially in the form of #:log-file, which would redirect a daemon’s
stdout/stderr to a file, and also in the form of “let’s hope the daemon
talks to syslogd”.  Here I was able to trivially add logging, such that
anything spawned by ‘fork+exec-command’ is logged, with timestamps and
all (the logger is a fiber that calls ‘read-line’ in a loop, which is
automagically non-blocking—delightful!).  The next fun step (besides
logging) will be adding inetd-style and/or systemd-style “socket
activation”.

Fibers is used in a single-threaded fashion, which is the main
constraint for shepherd since it forks.  That also means that fibers
cannot be preempted, so it’s fully cooperative scheduling.

There’s one catch: Fibers is currently Linux-only.  The good news is
that work has been done to port it to other kernels via libevent².
Until it is merged, we could keep using the Shepherd 0.8 on GNU/Hurd.

I’ve done some Guix System testing in VMs and didn’t notice any major
issues.  I’d like to merge that branch in ‘master’ and to eventually
release it as 0.9.0 (with or without socket activation, we’ll see.)
Hopefully, we could be running it within a couple of weeks.

Thoughts?

Ludo’.

¹ https://github.com/wingo/fibers
² https://github.com/wingo/fibers/pull/53