[epsilon-devel] Bootstrapping away from Guile: harder than it looks

[Luca Saiu]

Hello.  First things first, my usual disclaimer: Guile is awesome and I
really really like it; but epsilon is a different language, and has to
stand on its own.

The native epsilon1 REPL I shown in my previous message is fully
functional; it doesn't depend on Guile, and in fact I've already shown
how to run it on top of on a non-Guile image interpreter.  So what's the
problem?  Bootstrapping seems done.

Unfortunately it's not.

The problem is that the generated image /tmp/repl was built starting
from a memory state built with guile+whatever: we loaded a thin
temporary epsilon0 implementation and some more things such as e1:define
into Guile, and then defined all the essential runtime data structures,
and unexeced from that state.  That was "fair" from the bootstrapping
point of view: what guile+whatever loaded was only *source code*, in
textual form; we can accurately describe it as "the preferred form of
the work for making modifications to it", as per the GPL definition of
source code.

What would happen if I threw away all the Guile bits in epsilon and
started distributing the /tmp/repl generated image instead?  The
unexeced image wouldn't be source code, but there would still be the
source code as well: core.e, epsilon1.scm, and so on.  The problem is
that it is currently not trivial to generate a modified version of the
repl image from a previous version of it plus the source code.

This concern is not academic: it's very possible that in the future I
decide to change the representation of symbols (which may need more
fields) or of s-expressions (which could be made more efficient).  It
would still be possible to do that by loading /tmp/repl and then modify
its runtime data structures (even if that would be very delicate and
error-prone), but certainly a simple modification of the sources, say
core.e, would *not* suffice.  And I definitely want that.

One reason why it's difficult to reason about bootstrapping with
tombstone diagrams in epsilon is that the system works by
self-modification.  I don't have programs that produce other independent
programs: rather a program modifies itself and then, if it survives
self-surgery, it unexecs into a new image.  This way of working is
certainly powerful, but it's not always the right strategy for
re-defining fundamental structures and behaviors.

What's the right way?  I think I've found the right solution, but some
details are still fuzzy in my mind.

A generated interactive REPL such as /tmp/repl is certainly useful to
distribute; and I will distribute it.  But in order to let it load
incompatible versions of the core system, the image must be able to load
source files without breaking its own runtime structures: we need a
clear separation between "meta" state environments (in the sense of §2),
the "target" state environments; this is exactly what happens with
guile+whatever, and what makes bootstrapping on top of it easier.

Unexecing works by serializing what I call the symbol table --- Which,
by the way, might not the best name, as I noticed explaining the problem
to Alfred.  What I call "the symbol table" is symbol:table, which is to
say the table of symbols.  The symbol table is not a mapping from
symbols to values, but from names to symbol: it's the data structure
used to intern symbols.

My claim here is that we need a clear separation between a "meta" and a
"target" symbol table: one table will contain the running system runtime
structures, the other will contain runtime structures for the new system
image which will be dumped later by unexec.  The two tables are distinct,
and they are allowed to contain associate two different symbols to the
same name, thus locally violating the symbol-uniqueness invariant.

This idea has another important advantage: it will make it easier to
implement a personality built directly on top of epsilon0 without
sharing any of the epsilon1 runtime.  José made me notice that while I
often mention this possibility, I never describe any practical way of
achieving the goal.  His criticism is founded: the idea was very sketchy
in my mind as well, before I thought of this double-symbol-table
solution.

I've added a new git branch called "secondary".  In it I will modify the
system so as to accept definitions in a "secondary" symbol table, the
goal being to update core.e and epsilon1.scm to modify a "secondary"
table, before unexecing the secondary table into the "primary" one in an
image file.  Only after accomplishing that, I will finally be able to
claim in good conscience to have fully replaced guile+whatever with a
native epsilon1 system.  Then I'll make a release, and then implement
the stack introspection features discussed with Basile.

As usual I'll be very interested in your feedback.

Until the next time,

-- 
Luca Saiu
Home page:   http://ageinghacker.net
GNU epsilon: http://www.gnu.org/software/epsilon
Marionnet:   http://marionnet.org