Re: [Gcl-devel] Roadmap

[Camm Maguire]

Greetings, all, and thanks for your feedback!

Just to clarify one thing here -- I'd like to use the 2.4.x numbering
to refer to stabilization of Dr. Schelter's work.  I think 2.5.0
already should have new capabilities, as it does currently to my
understanding.  But this does remind us that we need to port a few of
the bug fixes currently in cvs and release a 2.4.2.  Would anyone like
to volunteer a patch, taken from a subset of 'cvs -rdiff Version2_4_1
HEAD' or whatever the cvs command would be?  I know of one clear error
which affected bignums.  In addition, we have yet to fix 'declare
integer' to add gmp support.  I've worked on this a bit.  I think we
should ensure that 2.4.2 will build maxima.

As for 2.5.0, I'm just summarizing the results here, but it seems the
consensus is that we should adopt the following:


Pre-2.5.0:

Major:

1) Clean builds with -Wall
2) resize default memory parameters and stack sizes
3) Check licensing
4) Incorporate pcl/clcs in some fashion (i.e. perhaps as loadable .o
        files)

Minor/Optional:

5) Whatever architecture patches I can get working in time.
6) compressed info file reading
7) working defsystem 
8) clocc tests at build time.

Post-2.5.0 (with comments):

1) Boehm gc option -- As I get more used to the code, the more
   impressed I am by various gcl features, which I think we should
   bring to the attention of our marketing department :-).  

        One of these is the current mm design.  I had suspected some
   mm/gc bug when I ran into difficulty porting gcl to other Debian
   architectures, but this has never been verified.  I've found posts
   by Schelter testifying to there being no leaks.  Far beyond this,
   we have an *exact* gc system to my understanding, the only negative
   of which I've read about is its difficulty of implementation :-).
   For those non-gc experts like myself, this means that the entire
   heap is explicitly walked with every possible pointer location
   checked, as opposed to making assumptions from a perspective of
   ignorance as to the actual data structures occupying the heap.  

        In addition, at least on Linux ix86, the system tracks when
   each page is written to by marking them all PROT_READ and catching
   the SIGSEGV, then changing the perms to add PROT_WRITE and marking
   the page for later gc processing.  Unwritten pages can be gc'ed
   much faster.  This 'stratified' gc seems like a great idea, but I
   still do not understand everything.  

        Once more, all pages are marked to hold only a certain class
   of lisp objects.  Don't know exactly how this is exploited yet, but
   it certainly should have advantages over the alternative.  

        Finally, malloc/free *appears to work just fine* across gc!!!!
   I haven't completely verified this statement, but I think the way
   it works is like this: mallocs return a pointer to a contiguous
   block of memory, which is then 'consed' into a malloc list in the
   lisp heap.  This effectively protects it from gc, and *I think* the
   page type t_contiguous is spared relocation as other items undergo
   gc.  I used to think one had to trap mallocs and redirect to a
   static array if one wanted to preserve storage across function
   calls with malloc, but I've tested eliminating this redirect, and
   all works apparently perfectly.  I'm just about to axe the
   do_bfd_malloc I added before -- the structures which bfd_init
   mallocs internally are apparently protected for the duration of the
   program from corruption by gc!  

        If this is correct, we have the following tremendous advantage
   -- we can use third party libraries without caring if they malloc
   or not, potentially opening up a vast array of standard C library
   functionality to gcl.  My only hesitation in this conclusion is the
   fact that Dr. Schelter had forked the gmp code and replaced an
   internal malloc with alloca, saying it could cause a problem for
   gcl.  But others have simply linked against -lgmp and proceeded
   without issue.  Perhaps the malloc emulation was not completed when
   Dr. Schelter incorporated gmp??  We need some good bignum test code
   to explore this issue.  Vadim?

2) UFFI -- forgot to mention this in the roadmap, but it fits in as
   others have noted with the additional bindings I was thinking of.
   I wish someone in the know could explain to me why a system can't
   just generically access external libraries with a syntax identical
   to the Fortran/C syntax, and have the internals of the system block
   gc for the call and connect the right addresses together.  Having
   to write a lisp wrapper for every external routine is going to get
   lisp nowhere fast in terms of new users, it would appear.  I know
   my idea is naive, but I still don't see the issue in principle.

3) external libgmp support -- as mentioned in 1) we may effectively
   already have this.  I just wanted to mention that what I'd like to
   do, at least for an installable system package like the Debian
   package, is to compile the gmp as shared libs for each
   subarchitecture relevant to the general arch, and install these
   libs in such a way so that gcl would use the fastest appropriate
   one given the running CPU.  The method of doing this has already
   been worked out for atlas on Debian, so should be straightforward.
   And I'm guessing that this would just about make gcl as fast as
   possible for bignums with minimal effort to the user.

4) ecls issues -- I'll try to fire off a note at some point to the
   developers. 

5) Makefiles -- moderately important, straightforward, tedious, adding
   nothing new in terms of functionality.

6) 64bit ports -- I feel we need to be ready for this.  Luckily, there
   is an alpha-osf1.h already.  
        
        This brings up a related issue: fasloading.  In the world of
        lisp, people seem to find the ability to dynamically load and
        run .o files an extremely useful, distinctive reason to use
        lisp in the first place.  And gcl certainly has the hooks to
        do it fast, though portability suffers.  The most portable way
        to do this in the old code appears to be with dlopen.  The
        disadvantage here is that one cannot save the system image
        with the loaded files, at least if Dr. Schelter's faq is still
        accurate.  The bfd work I've been doing would appear to extend
        current x86 behavior to *many* systems portably, at no cost in
        speed, but perhaps some increase in memory.  Here are the
        choices we face:

        1) No one cares about anything but x86, put this on the back
           burner. 
        2) People like to load, but don't care about saving the image,
           and are content to link the .o files explicitly if they
           wish.  Use dlopen for portability.
        3) People like this loading to be as fast and small as
           possible -- put the reloc information by hand into
           sfaslelf.c as Dr. Schelter did.
        4) People want the speed, but the size is not so critical.
           Use bfd everywhere, and take advantage of a probable speed
           increase as the symbol lookup proceeds through a hash table
           instead of a sorted binary tree.

        Just to report here -- Daniel Jacobowitz's suggestion on
        clearing the cache worked perfectly on ppc.  We now have a ppc
        port just as good as the others.  But everywhere expect on
        ix86, I'm still running into a maxima compilation problem.
        The let macro defined in the build gets corrupted to "let."
        somewhere. 

7) Rainer's check -- maybe keep this out of the distribution, but I
   need it to narrow down the issues I'm seeing in 6)

8) Performance -- 
        a) someone needs to comment on the *speed* of the current gc,
        as opposed to the number of gc's typically encountered in
        normal use.  I've heard conflicting reports, and would love
        some numbers.

        b) I feel we could easily access ISA extensions for bignums,
        and possibly also matrix operations via blas/atlas, but I'm
        not really sure if anyone particularly cares.

        c) Over the very long term, one could use gperf to make a
        perfect hash table for the saved_gcl image to be used in
        loading, lex/yacc to build a very fast lisp reader/parser,
        etc. 

Sorry for putting this all in one place.  I'm partly writing this to
provide a reference for our future work.  Comments/ideas of course
most welcome.

Take care,

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah