[MIT-Scheme-devel] A tale of four builds.

[Matt Birkholz]

The new rules in src/Makefile.in change our native build process
somewhat.  I traced through them, following along in my build log, and
came up with this Developers' Summary.

* Riastradh Build

The host syntaxes/compiles the new cref, then the new runtime and sf
(using the new cref).  The target cold loads the new runtime and dumps
it, then loads the new sf and syntaxes the new compiler.  The host
compiles the syntaxed compiler.  The target loads and dumps the new
compiler band, which is used to compile the rest of the system(s)
(except that the host compiles the star-parser?).

The result is an all.com band and loadable options containing a
host-compiled cref, runtime and sf; everything else is compiled by the
new compiler (except star-parser?).  This avoids compiling anything
twice, using Stage.sh, or tossing anything -- things that the
cross-compiles do with abandon.

The mix of host- and target-compiled subsystems assumes the new and
old machines are largely compatible.  The current change in s-code
types means the host-compiled .bcis contain surprises for the target
machine.

* LIAR/C Build

I traced through a liarc build too, a full-on cross-compile, and came
up with the following summary.

The host syntaxes/compiles the new cref, runtime, sf and star-parser
(using the old cref!).  The host cold loads the new runtime and dumps
an x-runtime band, then loads the new sf and syntaxes the new LIAR/C.
The host compiles the syntaxed compiler, then loads and dumps an
x-compiler band.  The build tree is then cleaned, stashing all
compiler products in STAGEX subdirs, preparing to cross-compile.  Just
the boot-dirs (cref, runtime, sf, compiler and star-parser) are
cross-compiled to C "bundles" (and the STAGEX subdirs tossed!).
Making distclean keeps the mit-scheme-c- distribution small.

The target (re)configures the build tree, c-compiles the machine and
bundles, installs the bundles in ./boot-root/lib/mit-scheme-c/lib, and
uses the new machine to cold load and dump a boot-compiler band.

The build tree is again distcleaned, tossing the cross-compiler's
products.  The new target uses the bundles and bands in
boot-root/{bin,lib}/ to compile everything, and tosses boot-root/.

* LIAR/svm Build

The LIAR/svm cross-compile is similar.

The host syntaxes/compiles the new runtime, sf, cref and star-parser
(using the old cref).  The host cold loads the new runtime and disk-
-saves an x-runtime band, then loads the new sf and syntaxes the new
LIAR/svm.  The host compiles the syntaxed compiler, then loads and
dumps an x-compiler band.  The build tree is then cleaned, stashing
all compiler products in STAGEX subdirs, preparing to cross-compile.
Just the boot-dirs are cross-compiled to svm1.

The target "finishes" the cross-compiled boot-dirs, then loads and
dumps a boot-compiler band.

The build tree is again cleaned, stashing the cross-compiler products
in STAGE0 subdirs.  The new target uses the boot-compiler band to
compile (NOT cross-compile -- by procedure) everything.

* LIAR/native Build

There is a fourth build process, a cross-compile that uses a LIAR/C
host to build a native system.  It is described in etc/make-native.sh.
It is the process used to create the first 64bit heap?  It does not
seem to be working currently.  I have some patches that get the
process through compile-boot-compiler.sh, but the new cross-compiler
craps soon after (because of a surprise in cref/make.bin?).  I traced
through the build process anyway.

The host syntaxes/compiles the new runtime, sf, cref and star-parser
(using the old cref).  The host cold loads the new runtime and disk-
saves an x-runtime band, then loads the new sf and syntaxes the new
LIAR/native, compiles it to C code, loads it and dumps an x-compiler
band.  The build tree is then cleaned, stashing all compiler products
in STAGEX subdirs, preparing to cross-compile.  Just the boot-dirs are
cross-compiled to native code (or svm1?).

The target "finishes" the cross-compiled boot-dirs, then cold loads
and disk-saves a boot-compiler machine and band.

The build tree is again cleaned, tossing the cross-compiler's products.

Now the build tree is (re)configured(?) and the machine (re)built?

The new target machine uses the boot-compiler band to compile (NOT
cross-compile -- by procedure) everything.

* Cross Compiling

The general pattern in the three cross-compiles is:

  1. Build the cross-compiler on the host.
  2. Cross-compile the boot-compiler on the host.
  3. Build the boot-compiler on the target.
  4. (Re)Compile everything with the boot-compiler.

Step 1 is the trick involving host-compiling the new runtime, sf, cref
and star-parser, then syntaxing the new compiler with the new runtime
(on the host!) and host-compiling the result.  It would be more
straightforward if the native and cross compilers could exist in the
same world, separately.

Step 2 assumes the target can fasload anything the cross-compiler
fasdumps (with finish-cross-compilation), but this is not always
possible, as when the s-code types changed recently.  The cross-
compiler needs a different (portickle? targeted?) fasdump.  The LIAR/C
compiler accomplishes this by dumping Scheme constants as C code that
rebuilds them in the target heap -- a fasdump targeting C.

Steps 3 and 4 may happen on a physically different target machine.
The LIAR/C build stops after Step 2 and prepares an mit-scheme-c-
distribution.  The other cross-compiling builds do not stop; they
assume the host and target Scheme machines are on the same physical
machine.

The Riastradh Build is very like Step 1, building a cross-compiler.
It adds the refinement of loading the new cref (but not star-parser?)
on the host before syntaxing and analyzing the new runtime and sf.
And it uses the target instead of the host to syntax the new compiler,
avoiding the need for a cross-runtime -- a new runtime on an old host.

* Scheme48ish Cross Compilation

The Scheme48 cool way to build would address the separation of host
(old) and target (new) compilers.  Our cross-compilation achieves that
by syntaxing the new compiler without the old compiler present, and
hoping that the old compiler can compile the resulting syntax.  The
new machine could interprete the resulting syntax, and thus compile
the compiler, but... that takes too long.

A Scheme48 cool AND cross build would separate old and new compilers,
loading the new compiler into the host as a separate cross-compiler --
into a (cross compiler) package linked to a (cross runtime) which
would, when necessary, contain a little training to simulate a new
heap in the old host.  Cross-compilation would cross-fasdump .mqocs
that the new machine's fasload can handle without finishment.  The
cross-fasdumper would be host code expecting simulated new-runtime
types and could bridge (be trained for) any change in heap format.

The resulting .mocs could be simply renamed .coms on the target.  The
new machine would load them into (compiler), (runtime), etc. packages
and pretend they were compiled by (compiler) on the new machine -- not
by (cross compiler) on a host machine.  Or they could continue to be
the mostly-native boot-compiler, tossed after it re-compiles
everything.  Or we could just check that there is no difference
between the .mocs and the boot-compiler compiled .coms (and the .coms
compiled by the compiler built by the boot-compiler?!).

* Coming At You

I found star-parser being compiled late in the build.  I think there
is a missing dependency that would get it pre-loaded in the host like
cref, before syntaxing the new runtime.  Thus a new runtime could use
a new star-parser operator, a situation which recently caused
problems.  Better yet: star-parser should be in runtime/ where it can
be loaded before it is used.

I don't know about converting compile-svm.sh to a set of
parallelizable makefile rules.  The need for Stage.sh makes it... not
obvious to me.  My only example is the pair of scripts
make-lairc-dist.sh and make-liarc.sh.

I would fix make-native.sh EXCEPT that I am not sure how useful it is
to build native systems on a LIAR/C host.  LIAR/C does not seem to be
a general-purpose system.  Its fasdump-to-C mechanism has limitations?
But it is currently our only way across the 32/64bit divide?

I would work on a cross-fasdumper, but I will need some really cool
heap format changes to motivate me... "really cool" like eliminating
SEQUENCE-3.

LIAR/C and separate compilation can stay limited, bootstrapping
techniques.  I drool for a modern luser's "just load it" build
abstraction where compilation is just part of loading/evaluating.
Autoloading pre-compiled "options" is so... last millenium.

If there is any objection to moving the star-parser into runtime/, now
is the time to stop me. :-) And if there is any pressing need for a
working make-native.sh, now is the time to ask.

Yours In Scheme,
-Friar Puck

Just for reference, here are my tracings of the four build processes.

* Riastradh build, detailed trace

        make
          make lib/runtime.com
            make compile-runtime
              make syntax-runtime
                make compile-cref
                  host: (compile-dir "cref")
                host: re-load (new) cref, then (load "runtime.sf")
              host: (load "runtime.cbf")
            make microcode/scheme
              make compile-microcode
                (cd microcode && make all)
            target: cold load new runtime, (disk-save "../lib/runtime.com")
          make lib/all.com
            make compile-compiler
              make syntax-compiler
                make compile-sf
                  make compile-cref, up-to-date
                  make syntax-runtime, up-to-date
                  host: re-load (new) cref, then (compile-dir "sf")
                make lib/runtime.com, up-to-date
                make microcode/scheme, up-to-date
                target: load new sf, then (load "compiler.sf")
              host: (load "compiler.cbf")
            make compile-edwin
              make bootstrap-toolchain
                make lib/compiler.com
                  make compile-compiler, up-to-date
                  make lib/runtime.com, up-to-date
                  make microcode/scheme, up-to-date
                  target: (load-option 'compiler) (disk-save "compiler.com")
                make microcode/scheme, up-to-date
              make syntax-edwin
                make bootstrap-toolchain, up-to-date
                make compile-xml
                  make bootstrap-toolchain, up-to-date
                  make compile-sos
                    make bootstrap-toolchain, up-to-date
                    target: (compile-dir "sos")
                  make compile-star-parser
                    make compile-cref, up-to-date
                    make syntax-runtime, up-to-date
                    host: re-load (new) cref, then (compile-dir "star-parser")
                  target: (compile-dir "xml")
                make syntax-win32
                  make bootstrap-toolchain, up-to-date
                  target: (load "win32.sf")
                target: (load "edwin.sf")
              target: (load "edwin.cbf")
            make compile-sf, up-to-date
            make lib/runtime.com, up-to-date
            make microcode/scheme, up-to-date
            target: (load-options '(COMPILER EDWIN)) (disk-save "all.com")
          make compile-compiler, up-to-date
          make compile-cref, up-to-date
          make compile-ffi
            make bootstrap-toolchain, up-to-date
            make compile-cref, up-to-date
            target: (compile-dir "ffi")
          make compile-imail
            make bootstrap-toolchain, up-to-date
            make compile-sos, up-to-date
            make compile-star-parser, up-to-date
            make compile-edwin, up-to-date
            target: (load-option 'edwin) (compile-dir "imail")
          make compile-runtime, up-to-date
          make compile-sf, up-to-date
          make compile-sos, up-to-date
          make compile-ssp
            make bootstrap-toolchain, up-to-date
            make compile-xml, up-to-date
            target: (compile-dir "ssp")
          make compile-star-parser, up-to-date
          make compile-win32
            make bootstrap-toolchain, up-to-date
            make syntax-win32, up-to-date
            target: (load "win32.cbf")
          make compile-xml, up-to-date
        Done.

* LIAR/C build, detailed trace

        make-liarc-dist.sh
          ./Setup.sh
          configure --enable-native-code=c
          make liarc-dist
            etc/compile-boot-compiler.sh
              host: (for-each compile-dir '(cref runtime star-parser sf))
              host: (disk-save "x-runtime.com")
              host.x-runtime: (load "compiler.sf")
              host: (load "compiler.cbf")
              host.x-runtime: (disk-save "x-compiler.com")
              Stage.sh make X
            etc/c-prepare.sh
              host.x-compiler:(for-each c-compile-dir
                                        '(runtime star-parser sf cref compiler))
              Stage.sh remove X
            make distclean

        make-liarc.sh
          ./configure --enable-native-code=c --prefix=./boot-root
          make stamp_install-liarc-boot-compiler
            make stamp_build-liarc-boot-compiler
              make stamp_install-liarc-boot-bundles
                make stamp_compile-liarc-boot-bundles
                  make stamp_install-microcode
                    make compile-microcode
                      (cd microcode && make all)
                    make install in microcode/
                  make compile-liarc-bundle foreach LIARC_BOOT_BUNDLES
                    make bundle.so
                      make module.o ...
                        microcode/liarc-cc module.o module.c -I ...
                      microcode/liarc-ld bundle.so module.o ...
                make install-liarc-bundle foreach LIARC_BOOT_BUNDLES
                  make bundle.so, up-to-date
                  mkdir libdir/mit-scheme-c/lib
                  install bundle.so libdir/mit-scheme-c/lib
              etc/build-boot-compiler.sh
                target: cold load and disk-save lib/boot-compiler.com
            make install-auxdir-top
              install lib/*.com ./boot-root/lib/mit-scheme-c/
          make c-clean
          make distclean

          ./configure --enable-native-code=c
          make stamp_compile-liarc-bundles
            make stamp_liarc-compile-scheme
              src/etc/c-compile.sh boot-compiler
                boot: (c-compile)
                  (in-liarc... (compile-all-dirs c-compile-dir) ...)
            make compile-liarc-bundles
              make compile-microcode, up-to-date
              make compile-liarc-bundle foreach LIARC_BUNDLES
                make bundle.so
                  make module.o ...
                    microcode/liarc-cc module.o module.c -I ...
                  microcode/liarc-ld bundle.so module.o ...
          make build-bands
            etc/build-bands.sh
              target: cold load and disk-save src/lib/runtime.com
              target: load all and disk-save src/lib/all.com
          make clean-boot-root
        Done

* LIAR/svm build, detailed trace

        make all-svm
          make microcode/svm1-defns.h
          make compile-microcode
            (cd microcode && MAKE all)
          etc/compile-svm.sh
            host: load runtime.sf, sf.sf and cref.sf
            host: load runtime.cbf, sf.cbf and cref.cbf
            host: load star-parser/compile.scm
            host: disk-save x-runtime.com

            host.x-runtime: load compiler.sf, sf base/crsend
            host: load "compiler.cbf"
            host.x-runtime: disk-save x-compiler
            ./Stage.sh make X
            host.x-compiler: boot-dirs

            target: (finish-cross-compilation:directory ".")
            target: cold load boot-dirs, disk-save boot-runtime, boot-compiler
            ./Stage.sh make-cross 0

            target --band boot-compiler: (compile-everything)
          make build-bands
            target: cold load and disk-save src/lib/runtime.com
            target: load all and disk-save src/lib/all.com
        Done.

* LIAR/native[or svm] build on LIAR/C, detailed trace

        etc/make-native.sh
          ./Setup.sh
          ./configure --prefix=`pwd`/boot-root
          etc/compile-boot-compiler.sh
            host: (for-each compile-dir '(cref runtime star-parser sf))
            host: (disk-save "x-runtime.com")
            host.x-runtime: (load "compiler.sf")
            host: (load "compiler.cbf")
            host.x-runtime: (disk-save "x-compiler.com")
            ./Stage.sh make X
          cd compiler && make compile-liarc-bundle
          # Error: no compiler-unx.c, needed by compiler-init.c
          #
          # Should have been produced by cbf-conditionally compiling
          # (fasdumping to C) compiler-unx.pkd, e.g. by
          # c-compile-dir in etc/compiler.scm.

          etc/native-prepare.sh
            host.x-compiler: (native-prepare)
              compile-boot-dirs with compiler:cross-compiling? #t

          make compile-microcode
          target: finish-cross-compilation:directory
          make stamp_install-native-boot-compiler
            make stamp_build-native-boot-compiler
              make stamp_install-microcode
                make compile-microcode
                cd microcode && make install
              etc/build-boot-compiler.sh
                target: cold load and disk-save lib/boot-compiler.com
            make install-auxdir-top
              install lib/*.com ./boot-root/lib/mit-scheme-*/
          make c-clean
          make distclean

          ./configure
          make stamp_native-compile-scheme
            make compile-microcode
            etc/compile.sh boot-root/bin/mit-scheme --library boot-root/lib/...
              boot-compiler: compile-cref, runtime, star-parser, sf
              target: (disk-save "runtime.com")
              target: (load "compiler.sf")
              boot-compiler: (load "compiler.cbf")
              target: compile-remaining-dirs
          make build-bands
          make clean-boot-root