Re: [Pika-dev] Systas eval.c

[Tom Lord]

    > From: Matthew Dempsky <address@hidden>

    > First, holy crap...  Tom, do you actually understand what's going on
    > in there?!

Yup.  I guess you're talking about the eval in Systas and I can tell
that just from the "holy crap" because that's my reaction too.  It is
a stunning piece of code.  Aubrey and Radey are hard-core, old-school
(not old, old-school), bad-ass hackers.


    > (ugh, tags.h gives me a head ache)

Yeah.  I think the best name I found for it is "staggered tags".

Basically, you can "get" eval once you get all the tag punning that's
going on and once you get the incremental translation stuff that's
going on.

Systas suffers from a deficiency: there's vestigial (slightly rotted,
never especially useful) support for an interactive debugger in
there.   You can pretty much just ignore all that.

And another deficiency: modern versions of SCM (contrasted with the
one Systas is forked from) have a special allocator and GC for
environment frames.   Since they are "almost stacklike" and references
to environment frames "leak" to the rest of a program only under very
specific circumstances (such as closure formation), modern SCM
allocates them stackishly and uses a copying collector to manage them
separately from the rest of the heap.   If an environment frame is
captured then it's copied to the general heap where mark/sweep takes
over but most leaf-node-in-call-graph frames get allocated and
collected very cheaply.

The biggest problems with SCM's eval are code size, excessive
indirection, and its interaction with SCM's mark/sweep collector.  The
internal form is large -- basically as large as the source s-exps.   A
bytecode representation would use /2 or /4 as much memory.
Sequential program steps in SCM (and Systas) are recorded as cons
pairs so that "increment the program counter" becomes "CDR" -- that
creates a pretty high lower-bound on memory traffic.   Finally, both
of these facts interact poorly with mark/sweep GC because all of the
(usually static) code must be traced at every GC cycle.


    > Okay, now that I've got that out, looking at the code I can see where
    > a lot of the ideas in some of Pika's design notes come from (leaving
    > DO as a semi-built-in), but with all the hairy bit manipulation stuff
    > (ugh, tags.h gives me a head ache) how much of eval.c can we actually
    > plan on using?  (FFI style issues aside.)

I do not plan to use eval.c or (for the most part) it's techniques.
I most especially do not plan to recreate the very, very hairy tagging
system of SCM.   Those things are works of beauty.  They are gems.
But they are a little dated (we don't have to work that hard anymore),
hell and a half to maintain, and a little dated again (they optimize
for a different class of hardware than we have).

See the soon-to-be-announced furth project for where I think Pika's
eval can come from.

    > Also, I know Tom's a big fan of SCM-style evaluation (rewriting sexps
    > through evaluation) because of its low latency properties, but Pika
    > seems interested in supporting a byte-code evaluating virtual machine.
    > It seems like these are conflicting implementation ideas aside from
    > something like having both virtual-machine-pika and
    > tasty-low-latency-pika executables, but maybe I'm missing some complex
    > control shifting mechanism?

Watch furth.

-t