[epsilon-devel] GNU epsilon café update: usabili ty improvements

[Luca Saiu]

During these last weeks I've made epsilon friendlier to common users,
who don't intimately understand the intricacies of the whole
bootstrapping-from-Guile process; I've also greatly improved the
compiler interface.

This message will be a quick tour, mostly focused on user-visible
changes.

Please get or update the epsilon sources from git (master branch), then
configure and build the obvious way -- Personally I don't build in the
source directory, but doing that is probably easier to beginners.  No
user-visible change up to this point.

--8<---------------cut here---------------start------------->8---
address@hidden ~/tmp/epsilon]$ ./autogen.sh
  [..output..]
address@hidden ~/tmp/epsilon]$ ./configure
  [..output..]
address@hidden ~/tmp/epsilon]$ make
  [..output..]
address@hidden ~/tmp/epsilon]$ 
--8<---------------cut here---------------end--------------->8---

At this point you need to bootstrap from Guile, which used to be clunky
and complicated.  Now you can simply build the repl target with make,
and an epsilon1 (non-Guile) REPL will be built, bootstrapping or
re-bootstrapping when needed, and started.  Bootstrapping is
comparatively slow: it takes seven minutes on my Yeeloong, and still
nearly one minute on a fast machine.

--8<---------------cut here---------------start------------->8---
address@hidden ~/tmp/epsilon]$ make repl
  [..long bootstrapping output..]
GNU epsilon git-snapshot
Copyright (C) 2012  Université Paris 13
Copyright (C) 2012-2014  Luca Saiu

GNU epsilon comes with ABSOLUTELY NO WARRANTY; enter `,show no-warranty' for
details.  This program is free software and you are welcome to redistribute it
under the terms of the GNU General Public License, version 3 or later.  Enter
`,show license' or see the file named COPYING for the full license text.

> 
--8<---------------cut here---------------end--------------->8---

At that point you can play with the REPL.  It's run on a tagged runtime,
so you can print objects in color notation and failures are a little
more graceful -- you don't usually get segmentation faults, but the REPL
still exits at the first error.  I'll manage errors properly after the
first release, when I get to implement some version of the
stack-handling epsilon0 changes discussed with Basile.

Right before the REPL banner you'll have noticed some output similar to
this:

--8<---------------cut here---------------start------------->8---
Defining the procedure fibo...
Defining the procedure test...
Defining the macro _2644...
--8<---------------cut here---------------end--------------->8---

This comes from bootstrap/scheme/scratch.e, which is automatically
loaded from the interactive REPL at startup.  The effects of scratch.e
are intentionally *not* included in the unexeced image: scratch.e is a
nice place to write and test experimental changes, without requiring a
slow bootstrap.

At this time scratch.e contains a test procedure accepting a fixnum
parameter n, which prints a table of Fibonacci numbers from 0 to n - 1.

--8<---------------cut here---------------start------------->8---
> (test 6)
fibo(0) = 0
fibo(1) = 1
fibo(2) = 1
fibo(3) = 2
fibo(4) = 3
fibo(5) = 5
> 
--8<---------------cut here---------------end--------------->8---

test can be a reasonable micro-benchmark to test procedure call
efficiency, when supplied with a parameter in the neighborhood of 30.
We'll get to that later.

The banner also mentioned the REPL command ,show which displays the GNU
GPL.  Its internal implementation is interesting, and shows how epsilon1
is becoming a powerful language.

,show is a prefix, just like ' , ` , , and ,@ .  A prefix followed by
exactly one s-expression reads as some prefix-specific s-expression:
for example 'E reads like (quote E), and ,@E reads like
(unquote-splicing E).  In the same way ,show copying is expanded into a
macro, as you can see by printing it as an s-expression with fio:write
(the epsilon1 version of printf in C or format in Lisp).  Notice that we
also need a quote to make a literal s-expression, and that se tells fio
to print what follows as an s-expression -- since epsilon1 is an untyped
language, you need to supply that information:

--8<---------------cut here---------------start------------->8---
> (fio:write "',show copying is read as " (se ',show copying) " .\n")
',show copying is read as (_1456 copying) .
--8<---------------cut here---------------end--------------->8---

The automatically-generated symbol, in this case called _1456, is a
macro name.  What does the macro do?

--8<---------------cut here---------------start------------->8---
> (debug:macroexpand '(_1456 copying))
[call repl:show [call sexpression:make 6₂₈₅₆₁₃ 0x3051468[0x2ff98c8[7 99 111 112 
121 105 110 103] 0 127 0 0 0 0 0 0 0 0]₂₈₅₆₁₄]₂₈₅₆₁₅]₂₈₅₆₁₆
--8<---------------cut here---------------end--------------->8---

(_1456 copying) simply rewrites into an epsilon0 procedure call, taking
an s-expression as its only parameter; the s-expression is the s-symbol
copying (tag 6 (symbol), and the symbol at 0x3051468, which contains the
string "copying" at 0x2ff98c8 as its first element).  repl:show is an
ordinary procedure, dispatching on the value of its parameter and
printing something.  repl.e contains this code:

--8<---------------cut here---------------start------------->8---
(e1:define (repl:show what)
  (e1:case (sexpression:eject-symbol what)
    ((license copying gpl)
     (fio:write (st repl:gpl-text) "\n"))
    ((no-warranty warranty)
     (fio:write (st repl:no-warranty-text) "\n"))
    (else
     (e1:error "Unknown ,show argument\n"))))
--8<---------------cut here---------------end--------------->8---

Even the implementation of the REPL-command definition functionality is
surprisingly elegant: see the "REPL commands" section in repl.e.  It's
now very easy to add new REPL commands -- which actually resolve to
procedure calls, usable from anywhere in the code and not necessarily at
the top level.

There's one last beautiful hack in ,show .  You'll have noticed the
globals repl:gpl-text and repl:no-warranty-text in the repl:show
definition above.  They are bound to big strings, which I definitely
don't want to embed in my sources; but I have the complete texts in two
files, in the source distribution: COPYING and NO-WARRANTY.  The
solution is to have a *macro* accepting a file name as a parameter,
which expands to a *literal* string holding the file context, character
by character.  After unexecing the REPL such texts will automatically
become part of the image, without any residual dependency from text
files.  The full implementation of this feature is only 13 lines long;
see epsilon1.scm, the section named "Read the whole content of a file
into a byte vector".

Back to the test procedure.  Run this just to get a feel of how long it
takes, which should be least a few seconds:

--8<---------------cut here---------------start------------->8---
> (test 33)
fibo(0) = 0
fibo(1) = 1
fibo(2) = 1
fibo(3) = 2
fibo(4) = 3
fibo(5) = 5
fibo(6) = 8
fibo(7) = 13
fibo(8) = 21
fibo(9) = 34
fibo(10) = 55
fibo(11) = 89
fibo(12) = 144
fibo(13) = 233
fibo(14) = 377
fibo(15) = 610
fibo(16) = 987
fibo(17) = 1597
fibo(18) = 2584
fibo(19) = 4181
fibo(20) = 6765
fibo(21) = 10946
fibo(22) = 17711
fibo(23) = 28657
fibo(24) = 46368
fibo(25) = 75025
fibo(26) = 121393
fibo(27) = 196418
fibo(28) = 317811
fibo(29) = 514229
fibo(30) = 832040
fibo(31) = 1346269
fibo(32) = 2178309
> 
--8<---------------cut here---------------end--------------->8---

As usual you can unexec and run the computation from the unexeced image;
this is nothing new.  Let's call the unexeced image /tmp/q.u :

--8<---------------cut here---------------start------------->8---
> (e1:unexec "/tmp/q.u" (test 33))
> 
--8<---------------cut here---------------end--------------->8---

Now, the following *is* new: you can use the compiler with the exact
same interface of unexec, and make a standalone native-code executable:

--8<---------------cut here---------------start------------->8---
> (e1:compile "/tmp/q" (test 33))
> 
--8<---------------cut here---------------end--------------->8---

e1:compile uses the native compiler, if available; right now I support
x86_64 and MIPS.  If your architecture is not directly supported,
e1:compiler generates and compiles C code.

The compiler itself is very simplistic and quite ugly; it even predates
fio:write-to.  It generates stack-based code, where each temporary has
to be saved to a stack slot and then re-read when used.  All primitive
calls involve calls to a C function, even for trivial primitives such as
fixnum:+ .  What I want to say is that it's *trivial* to do better than
the current compiler, but the generated code has already quite decent
performance.

At this time the compiler only supports static, in the sense of
non-self-modifying code.  Of course I will extend it to the general
code, but I will continue to support more efficient compilation for the
static case.

Let's get out of the epsilon1 REPL with C-d, and see how fast the
generated code runs.

You can compare it with the (interpreted) code from the unexeced image.
Of course you can omit the output redirection if you want to see the
Fibonacci number table.

The untagged version should be more or less as fast as the one you ran
interactively on the REPL -- possibly just a little slower because of
the time needed to undump the image.  The compiled version will be
*much* faster.

--8<---------------cut here---------------start------------->8---
address@hidden ~/tmp/epsilon]$ time bin/epsilon-image-interpreter-smob /tmp/q.u 
&> /dev/null

real    0m29.046s
user    0m29.020s
sys     0m0.040s
address@hidden ~/tmp/epsilon]$ time bin/epsilon-image-interpreter-tagged 
/tmp/q.u &> /dev/null

real    0m13.069s
user    0m13.064s
sys     0m0.012s
address@hidden ~/tmp/epsilon]$ time bin/epsilon-image-interpreter-untagged 
/tmp/q.u &> /dev/null

real    0m8.861s
user    0m8.864s
sys     0m0.000s
address@hidden ~/tmp/epsilon]$ time /tmp/q &> /dev/null

real    0m0.318s
user    0m0.316s
sys     0m0.000s
address@hidden ~/tmp/epsilon]$ 
--8<---------------cut here---------------end--------------->8---

More than 20 times faster, even with my crude compiler!  We're still far
away from the performance of code generated by GCC, but this is already
quite acceptable.

Of course you can still run epsilon1 on guile+whatever.  This is mostly
for myself, while I work on bootstrapping on top of epsilon1 only and
away from Guile.  Of course you still need to use the e1:toplevel Guile
macro, and the system is much slower.  However at least you can easily
enter the guile+whatever REPL, letting make bootstrap when neeed.

--8<---------------cut here---------------start------------->8---
guile> address@hidden ~/tmp/epsilon]$ make guile+whatever-repl
  [..short and fast loading..]
guile> 
--8<---------------cut here---------------end--------------->8---

That's it for today.  Now I'll work on the bootstrapping thing.

Regards,

-- 
Luca Saiu
Home page:   http://ageinghacker.net
GNU epsilon: http://www.gnu.org/software/epsilon
Marionnet:   http://marionnet.org