Re: [Pika-dev] Lexer fix

[Tom Lord]

    > From: Matthew Dempsky <address@hidden>

    > I patched the lexer so it will read characters like #\M-% now
    > (rather than just #\M- like it would originally).

I'll catch up with that.

    > What's the current plan regarding implementation of numbers?  I figure
    > small fixed size integers will be packed into the t_scm_word like
    > other Lisp implementations do, but are you planning on reimplementing
    > a library like libgmp or just link to it?

As you know (but I'll repeat for the list) the implementation is
roughly divided into src/scm/reps and src/scm/libscm:

The libscm part is "generic" -- that code should keep working no
matter what changes or forks are made of pika.

The reps part is "implementation specific" -- that code "knows about"
the representations of objects, the GC, and so forth.

The idea is that you can do something like implement a new tagging
strategy or a new GC by swapping out the reps part, leaving everything
else intact.

And, as you've observed privately, a little bit of code has leaked
into reps that really should be moved to libscm -- that'll get fixed
eventually.  Currently the line counts are:

    reps:   2768
    libscm: 3644

but that ratio can be improved a bit.

The idea of using fixnums packed into t_scm_word is a reps issue, not
a libscm issue.  Yes, that is the plan (the tags are already reserved
for it, as you can see) but the client interface won't know about that
detail.

The plan for numbers is roughly this:

  ~ start to define a libscm (i.e. generic) interface to numbers.

    In that interface, you'll be able to ask "is this value a number?"
    and "can i (precisely) convert it to a C integer type?"  and, if 
    so, "what's its value?"

    And you'll be able to ask "can i convert this C integer value to a
    Scheme value?"  and if so "what is that value?"

So, with that as a starting point, yes, fixnums will come first.

Inexact reals mapping to and from C floats and/or doubles will work
similarly at the libscm level.  At the (current implementation of)
reps level, as you can see from the tagging scheme, I plan to provide 
heap-allocated values holding a C double.

As for bignums: there's really a design choice here and I'm not sure
yet how to make it.  The options I see are:

  1) keep it entirely at the Scheme level.   You can have bignums,
     but you have to operate on them as Scheme values.   If they
     happen to internally be GMP numbers, that's just a private 
     detail of reps.

  2) encorporate gmp directly into the libscm (generic) interface.
     Yes, you _can_ operate on numbers just at the scheme level, 
     but you can also surface them as GMP.

I'm rather far from being a numerics expert, unfortunately.   I would
describe myself as a "poor numerics amateur".   With that caveat:

I'm inclined towards (2) for three (rather unscientific) reasons.  (a)
Torbjorn made a good impression on me when I met him.  (b) My sense
was that he was both qualified for and committed to a very long-term
project to keep tuning and extending GMP with great care, as a kind of
avocation if nothing else.  (c) I've almost never heard anything _bad_
about GMP and quite a bit that's good.

And one pragmatic reason:  if we regard GMP as a quasi-standard, then
surfacing it at the libscm level is useful.

Possibly against (2) are some things I haven't researched yet.
There's more to Scheme numbers than just bignums: rationals, complex
numbers, and so forth.  I'm not sure, at the moment, where GMP is on
these other types.  Possibly it will make sense to extend GMP in those
directions; possibly we'll wind up with a full Scheme numeric type
that can be operated on only using libscm interfaces (with a few
exceptions like integers, inexact reals, and bignums) -- or possibly
they'll all map into GMP-plus-extensions.



    > Either way, I was thinking about working on adding lexer support for
    > numbers (unless you think you will be needing it very soon, in which
    > case I'd recommend you write the core part and leave me some FIXMEs to
    > work on once it's Good Enough).

I think that letting me do a tiny core with plenty of #undef FIXMEs is
the way to go -- and that will liberate you to hack on it further to 
your hearts contents.   My abbreviated TODO list as of today says:

     strings
     numbers 
     gc
     environments, locals, bindings
     alpha-ids
     vm, procedures, transformers
     simple compiler


I think I will swap that around so that the first two items are
reversed.

-t