Re: IPC etc. (was: Future Direction of GNU Hurd?)

[William ML Leslie]

On Sun, 21 Mar 2021 at 05:09, Olaf Buddenhagen <olafbuddenhagen@gmx.net> wrote:
>
> Hi again,
>
> On Mon, Mar 15, 2021 at 07:58:33PM +1100, William ML Leslie wrote:
> > On Mon, 15 Mar 2021 at 05:19, Olaf Buddenhagen <olafbuddenhagen@gmx.net> 
> > wrote:
> > > On Thu, Feb 25, 2021 at 06:48:11PM +1100, William ML Leslie wrote:
>
> > > > I am still hopeful that someone will figure out how to do async
> > > > right at some point.
>
> > It's somewhat easy to stay as close to io_uring as possible, having
> > ring buffers for messages ready to be sent and messages to be
> > received.
>
> It's quite unusual to treat generic async IPC as the same as async
> I/O... Though to be frank, it actually is the same in the IPC approach
> I'm envisioning :-)
>
> The issue with io_uring however is that it's designed for the monolithic
> system use case, where all I/O is handled by the kernel. In a
> microkernel environment, rather than having a shared memory protocol
> between userspace processes and the kernel, the obvious approach is to
> have these implemented directly between the clients and servers
> instead...
>
> Of course that doesn't help if we want a unified I/O queue for requests
> to multiple servers: however, it's not immediately obvious to me that
> implementing a mechanism for that in the kernel (instead of client-side
> handling) is indeed a good idea...
>

Yes.  Specifically, the problem returns to one of synchronisation.  If
you want to send a message Y only after X has been received, you
either get to implement that logic in-process and incur the two
context switches to receive Y and queue X, or you can implement that
logic as part of the channel between two systems which means that you
can't have X come from a different server than Y is going to (because
how will it understand that ordering relationship), or you can have
the kernel enable the subsequent send.

> > We're a bit spoiled in the capability space, though.  Many of our
> > protocols include the ability to send a message before its target or
> > arguments become available, as well as the ability to only have a
> > message sent on receipt or failure of a previous message.
> [...]
> > The third solution is to add logic to the kernel to perform further
> > sends when a message is received, and complicating the kernel at all
> > is frankly a bit scary (and how should we time-bound that?).
>
> If we take that to the logical extreme, we get grafting, i.e. uploading
> arbitrary logic expressed in a safe language -- a concept that has been
> proposed in academia decades ago, but AFAIK hasn't made it into the
> mainsteam until recently, in the form of eBPF in Linux...
>

BPF is 14 years old, so it's not so new, but having basically an
interpreter in kernel space is an idea that certainly has made me
giddy.  Hopefully I can avoid it, but at this point, maybe not.

> Intuitively, it doesn't feel like accounting would be much different
> than for individually issued kernel requests?...
>

Maybe.  It's easy to forget that EROS was the extremely *reliable*
operating system, and that transactionality of message send/receive is
quite a beautiful guarantee.  In the async case, there's quite a few
more locks than usual that we need to acquire, but also drop again
without exiting the kernel, and we only have one other case where we
do this (and it's already very scary).  Anyway, once locks are
acquired, the transfer must complete.  In the case of large messages
or a large number of messages, it may be difficult to meet scheduling
requirements.  Anyway, it's a pretty abstract problem at the moment.

> However, I realised that the way I'm approaching IPC, it is actually
> quite natural to express the major use cases in the IPC protocol -- thus
> mostly eliminating the motivation for doing generic logic...
>

Nice work if you can get it.

> > I plan to do a little more on a per-process basis.  A few turns before
> > a process is scheduled, we make sure to page-in anything the process
> > is about to touch.  A capability provides the means for a process to
> > say which addresses it will soon access, in addition to the
> > instruction pointer.
>
> That's an interesting idea... Not sure though whether there are actual
> use cases for that: in theory, it could reduce the latency for processes
> that are resumed after having been idle long enough to get paged out --
> but only if they know in advance that they need to resume soon... I
> can't think of a situation where a process needs to react quickly to an
> event scheduled long in advance?
>

I've got plenty of use cases.

One is to make string transfer in Coyotos reliable.  You say "require
these pages to be resident before sending" and then the message send
will delay until the string transfer will complete without paging.
Particularly important with async IPC, where loads of messages may be
touching all sorts of pages.

Another is a GC heap walk.  Most operating systems get very confused
by GC and get to the point where they make no progress, because it
sees that pages were recently touched and so decides shouldn't be
paged out.  Having the GC declare where its fingers are and where they
are headed as the requirement for residency gives the power back to
the process.

> > An aside: I absolutely want to have optional orthogonal persistence
> > per-application.  Imagine having emacs up and ready to go the moment
> > you log into your machine.  Yes please.
>
> How would that work per-application? Don't we have to restore the
> application's environment (including capabilities) -- meaning it has to
> be more or less system-wide?...
>

CapROS (and presumably EROS, too) have a set of non-persistent
applications that most of the persistent processes depend on.  It
feels a little like how GNU shepherd and systemd pre-open sockets and
pretend the application is already available.

To Shap's objection: I say per-application because I mean that you
should be able to decide which applications take part in the
persistent state, not because each application is somehow checkpointed
individually.

> Either way: yes, I totally want the ability to seamlessly resume any
> activities (that do NOT include Emacs in my case :-P ) after logouts,
> power cycles etc. Indeed I consider it among the two or three most
> important must-have features of my design. (Maybe *the* most important
> one? It's hard to rank them...)
>
> However, I don't intend to implement this with orthogonal persistence
> based on preserving the entire memory image of processes.
>
> This type of persistence mechanism is very tempting, because it feels
> like it provides very desirable functionality with very little effort.
> The problem is at the edges, where it doesn't help: things like
> upgrading software; migrating part of the environment to a different
> system instance; recovering from crashes involving memory corruption...
>

It's not a statement of intent, but rather awareness that I'm not able
to boil the ocean.  Which is to say: I don't have the energy to
rewrite all the applications I use to use the SLS.  Perfection is a
bit beyond me at this point.

> Of course transparent orthogonal persistence doesn't *preclude* handling
> these situations: we just need to serialise all precious state to carry
> it forward explicitly... The thing is, once we have such a serialisation
> mechanism, why do we need the other persistence mechanism at all? Better
> make serialisation the sole persistence mechanism; ensuring it works
> really well, rather than just being a poorly maintained backup mechanism
> for special cases...
>
> (Shap will probably tell me that I got it all wrong or something: but
> the truth is that my conclusions on this matter haven't budged over the
> past 15 years -- and I can't imagine them budging over the next 15 :-) )
>
> > > I don't see a fundamental difference... Whether it's ext2, SLS, or
> > > some sort of RAM disk: in each case you need a driver that can be
> > > loaded by the bootloader?...
> >
> > It's just a matter of complexity.  The various pieces that implement
> > the SLS are less than 5000 lines, wheras libstore is over 7000 on its
> > own; libdiskfs 12000, and then libext2 on top of that.  But yes, it's
> > somewhat like an initrd.
>
> Why would that matter, though? You aren't limited in the size of the
> image loaded by bootloader, are you?...
>

Yes, incidentally.  I mean, obviously not on the machines I have that
run GRUB; but I want to run HURD on my router and my telephone and
some of the older tech I still use.

-- 
William Leslie

Q: What is your boss's password?
A: "Authentication", clearly

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.