[MIT-Scheme-devel] cleanly building the C back end

[Taylor R Campbell]

I think I have gotten all this mostly working, at least for the old
microcode.  With these modifications, it should be possible to prepare
a fresh CVS tree for clean, stand-alone distribution to other systems
like so:

% cd v7/src/
% ./Setup.sh
% ./configure --enable-native-code=c
% make c-prepare
% make c-clean

And then to build a complete Scheme system from a distribution tree
like so:

% ./configure --enable-native-code=c
% make c

I've put up a stand-alone distribution here; it seems to work on a
number of different systems:

<http://people.csail.mit.edu/riastradh/tmp/c-boot.tar.bz2>.

If you get an error while the build process running utabmd.sh, try
typing

% touch microcode/utabmd.bin
% touch microcode/utabmd.c

and repeating `make c'; I haven't figured out why this is happening.

So, these are all the changes I had to make...

* CREF OS Types

I had to fiddle with the CREF package description fasls to get exactly
the right files generated and statically linked.  Of the subsystems
runtime, sf, cref, compiler, and star-parser, all but star-parser's
are named exactly like the subsystem, and all but compiler's use all
three operating system types.

So my kludge was first to hack script.scm (which I'd like to rename to
`c-prepare.scm') so that it compiles to C exactly the right package
description fasls, and then to add a new file
microcode/makegen/pkds-liarc.scm which lists all of the pkd files that
need to be statically linked, and also to update makegen.scm so that
it includes these in `liarc-rules-1'.

The problem with this is that it hard-codes assumptions about what OS
types there are, and what filename suffixes they use.  Both
c-prepare.scm and makegen.scm must know about this, and pkds-liarc.scm
must know that the compiler subsystem generates only a Unix
description.

It would be nice to replace all this CREF stuff once and for all.  In
a few weeks, or perhaps before then, I really do intend to work on the
module system that I've been mumbling about for so long, but that's a
little outside the scope of getting the C back end to work.

For now, this kludge for CREF OS types will work, I think, although
it's rather fragile.

* N-Stage Build

In order to fully bootstrap from a fresh CVS tree, we need a natively
compiled compiler with the C back end.  Unfortunately, generating this
requires syntaxing and creffing the compiler's sources *without* the
native compiler loaded, because the package names would conflict, and
there's no way in a standard distribution to load only CREF and SF
without the compiler.

So first we must compile CREF and SF with the native compiler;
load the target CREF and SF into a compiler-free native system to
syntax the target compiler; and finally compile, without syntaxing or
creffing, the syntaxed target compiler using the native compiler.
After that, we load it all up and save a `boot-compiler.com' band,
which is a natively compiled band with SF, CREF, and LIAR with a C
back end.

Once we have boot-compiler.com, we can begin to compile the C sources,
which is accomplished by c-prepare.sh.  Then we clean up everything
but the compiled C code, putting the tree in a state suitable for
distribution.

In a distributed tree, we build all the C code and statically link the
runtime and compiler and such into the executable.  But there is no
debugging information for any of this, and what is statically linked
excludes Edwin and other useful systems, so we need to recompile,
using the statically linked microcode.  This is the longest part of
the process.  After it, everything should be built and ready for
running.

* Cleaning for C Distribution

At the moment, I'm using `make distclean' in the microcode followed by
some kludges to clean up a C-bootstrapped distribution tree.  First,
the symlinks in the `compiler/' subdirectory must be deleted, so that
reconfiguring won't lose when trying to create them again.  Then,
utabmd.bin and utabmd.c must be touched so that `make' won't try to
rebuild utabmd.bin from utabmd.scm with a non-existent system-
installed Scheme.  Finally, all fasl files constructed in the above
process must be deleted: *.bin, *.ext, *.com, *.bci, *.pkd, & *.fre.
However, *.c obviously must not be deleted.

I had to omit utabmd.c from MOSTLYCLEAN_FILES in the microcode
makefile to ensure that utabmd.c would remain in the tree.  I'm not
sure why it's deleted anyway.

I wonder whether it would be easiest to add a Clean.sh option to clean
for C.

* Compiling for Dynamic Loading

The parts of the system that are statically linked to the microcode
must be compiled (.c -> .o) with COMPILE_FOR_DYNAMIC_LOADING
undefined; everything else needs COMPILE_FOR_DYNAMIC_LOADING to be
defined.  config.h defines COMPILE_FOR_STATIC_LINKING by default if
the native code is C, so this is the default unless overridden,
irrespective of the conditional in liarc.h.  However, if LIAR is
invoking the C compiler, it is probably not for the sake of static
linking, since all the compilation for static linking happens due to
rules in the microcode makefile; therefore, I decided that it would be
easiest to put -DCOMPILE_FOR_DYNAMIC_LOADING in the switches that LIAR
passes to the C compiler in C-COMPILE in ctop.scm.

This is not ideal; I don't know whether it will work for all C
compilers.  Perhaps it would be better to put the switch in the
C-COMPILER-SWITCH-TABLE, but this seems to work for now.

* Portability Issues

(These are not necessary for getting the C back end working, but they
would be good to fix anyway.)

I stumbled across some problems on NetBSD (x86) and Solaris 9 (SPARC).
Although some parts of the system know about NetBSD, the configure
script doesn't set up MODULE_LDFLAGS correctly, and LIAR's C compiler
switch table has an entry specialized to NetBSD on x86-64, not just
NetBSD in general.  Since someone else tested the NetBSD build, and
since I don't have access to a NetBSD machine myself (are there any at
CSAIL that I can use?), I haven't been able to try any changes.  I
think more are probably necessary, but I'll worry about that later.

For Solaris 9, I had to change a number of things:

1. Comment out all of the various default declarations of `malloc'
   that are scattered throughout the source.  I didn't take the time
   to figure out what macro the compiler defines to indicate a Solaris
   environment which I could add to the conditional.  I wonder whether
   it might be better just to isolate those declarations to a single
   file.  The files I had to modify thus were:
     boot.c error.c ptrvec.c wind.c

2. Include <ucontext.h> in uxtrap.h.  I wasn't sure where to put it in
   there; I quickly grew lost among the cpp conditionals.

3. Detect sockets correctly.  I still haven't figured this one out
   yet; for the moment, it worked just to fix the SHUTDOWN-SOCKET
   primitive to wrap its body in SOCKET_CODE.

4. Pass the correct arguments to the linker.  GCC seems just to pass
   its arguments to `ld', which on my Sun machine anyway isn't the GNU
   ld, so it reads the argument `-export-dynamic' as something to do
   with the symbol `xport-dynamic'.  Omitting that argument seems to
   work to statically link the microcode, although, since this is
   merely a 333 MHz, it will be several more hours before I can be
   sure that the build worked.