Re: The Shepherd on Fibers

[Ludovic Courtès]

Hi!

Philip McGrath <philip@philipmcgrath.com> skribis:

>> Fibers is used in a single-threaded fashion, which is the main
>> constraint for shepherd since it forks.  That also means that fibers
>> cannot be preempted, so it’s fully cooperative scheduling.
>> 
>
> Would it be feasible for shepherd *not* to fork?

Sure, but that would make it a daemon supervisor that doesn’t launch any
daemon, which would make it less useful.  :-)

> Or only to fork in a way that cooperates with fibers?

A process that has multiple POSIX threads cannot fork reliably; that’s
the core of the problem (it’s a problem of POSIX and ‘fork’, not
specific to Guile and Fibers).

> Obviously forking is pretty ubiquitous, but I the 2019 paper "A fork()
> in the road"[1] fairly convincing in its argument that

Yes, I read it before.  We could have bindings for ‘posix_spawn’ but
it’s kind of ugly and limited.

> More concretely, preemption is a big benefit of fibers.

I’m not arguing against preemption, I like it.  :-)

However, as currently implemented in Fibers, we cannot rely on it,
though Maxime mentioned that it could use signals instead of a separate
thread, which would work.  We can always revisit that later.  I making a
first step here, and being cautious, but we can and should unleash our
creativity once we have the basics in place.

> Racket has a cross-platform C library, "rktio"[3], for accessing
> os-specific functionality. It was refactored into its current form in 
> 2017 as an early step toward Racket-on-Chez: while it happens to
> provide exactly the functionality Racket needs, it no longer is
> specific to Racket or any particular implementation thereof. That
> includes everything needed to implement the Concurrent ML system and
> nonblocking IO on a variety of OSes and kernels.
>
> In particular, it implements—IIUC primarily in
> "rktio_process.c"—abstractions (over `fork` or alternatives) to start
> a new process running something, with environment, stdin, stdout, and 
> stderror wired up ports in the sense of
> `current-{input,output,error}-port`, and use the Concurrent ML system
> to monitor its exit status, send it signals, etc. The Racket-level API
> is documented at [4].

Interesting.  I’m not sure this is directly translatable to Guile.
However, libevent support for Fibers should address this specific point,
so I’m not really concerned.

> Second, I'm a little uneasy about `unwind-protect`:

Your criticism is valid, though here it has a single user and it’s “good
enough” for that case I think.

Thanks for your feedback!

Ludo’.