[Gcl-devel] 2.6.2 benchmarks

[Camm Maguire]

Greetings!  Just finishing up the FreeBSD tests thanks to Mark Murray
-- should be very soon now.

I noticed in Dr. Schelter's original gcl distribution directory
(ftp://ftp.ma.utexas.edu/gcl/) a package of gabriel benchmarks.  Just
for fun I decided to run them with our 2.6.2, and latest cmucl (18e-9)
and clisp (2.33-2) on Debian sid.  The results are quite favorable to
gcl, so much so in some cases that I suspect that the cmucl case was
not adequately proclaimed, though the test attempts to do so.

All I did to get these to run is 

1) Apply the following patch to
        a) make the |si| package if it does not exist
        b) expand the runtime by 400 to avoid granularity on modern
        machines 
=============================================================================
diff -ru foor/make-declare.lsp gabriel/make-declare.lsp
--- foor/make-declare.lsp       1992-01-14 23:15:42.000000000 +0000
+++ gabriel/make-declare.lsp    2004-06-14 20:29:46.000000000 +0000
@@ -1,7 +1,9 @@
 ;; By W. Schelter
 ;; Usage: (si::proclaim-file "foo.lsp") (compile-file "foo.lsp")
 
-(in-package 'si)
+(unless (find-package "si")
+  (make-package "si"))
+(in-package "si")
 
 (proclaim (quote (optimize (compilation-speed 0) (safety 0) (speed 3)
                           (space 0)
diff -ru foor/makefile gabriel/makefile
--- foor/makefile       1998-09-02 22:45:39.000000000 +0000
+++ gabriel/makefile    2004-06-14 20:39:04.000000000 +0000
@@ -1,52 +1,51 @@
-# Makefile for running gabriel bench marks
-# Do 
-# make "LISP= lisp name "
-# except for CMULISP which requires
-# make "LISP=cmulisp" "QUIT=(quit)"
-
-
-
-FILES=  boyer browse  ctak dderiv deriv destru-mod destru  div2 fft-mod \
-       fft fprint fread frpoly  puzzle-mod puzzle stak \
-       tak-mod tak takl takr tprint traverse triang-mod triang
-
-REPEAT=1
-
-LISP=akcl
-RESULTS= times
-
-all: times
-       make -e "LISP=${LISP}" "FILES=${FILES}" "QUIT=${QUIT}" compile
-       @ echo >> $(RESULTS)
-       @ echo "--------------   SESSION   ------------------" >> $(RESULTS)
-       @ echo >> $(RESULTS)
-       @ echo "      " For $(LISP) Common Lisp  >> $(RESULTS)
-       @ date >> $(RESULTS)
-       make -i -e "LISP=${LISP}" "FILES=${FILES}" "QUIT=${QUIT}" test
-
-compile:
-       for v in $(FILES) ; do \
-       echo "(load \"make-declare.lsp\")(si::proclaim-file \"$$v.cl\")" \
-       "(compile-file \"$$v.cl\")" "$(QUIT)" | $(LISP) ; done
-
-
-test: times
-       for v in $(FILES) ; do \
-       echo "(load \"test-help.lsp\")(do-test \"$$v\" \"$(RESULTS)\")${QUIT}" \
-       | $(LISP); \
-       done
-
-clean:
-       rm -f core *.o *.lbin *.bin *.*fasl *~ *#
-
-times:
-       touch times
-
-tar:
-       tar cvf - *.cl *.lsp makefile  | gzip  -c >  gabriel.tgz
-
-
-
-
-
-       
+# Makefile for running gabriel bench marks
+# Do 
+# make "LISP= lisp name "
+# except for CMULISP which requires
+# make "LISP=cmulisp" "QUIT=(quit)"
+
+
+
+FILES=  boyer browse  ctak dderiv deriv destru-mod destru  div2 fft-mod \
+       fft fprint fread frpoly  puzzle-mod puzzle stak \
+       tak-mod tak takl takr tprint traverse triang-mod triang
+
+REPEAT=1
+
+LISP=akcl
+RESULTS= times
+
+all: times
+       make -e "LISP=${LISP}" "FILES=${FILES}" "QUIT=${QUIT}" compile
+       @ echo >> $(RESULTS)
+       @ echo "--------------   SESSION   ------------------" >> $(RESULTS)
+       @ echo >> $(RESULTS)
+       @ echo "      " For $(LISP) Common Lisp  >> $(RESULTS)
+       @ date >> $(RESULTS)
+       make -i -e "LISP=${LISP}" "FILES=${FILES}" "QUIT=${QUIT}" test
+
+compile:
+       for v in $(FILES) ; do \
+       echo "(load \"make-declare.lsp\")(when (fboundp '|si|::proclaim-file) 
(|si|::proclaim-file \"$$v.cl\"))" \
+       "(compile-file \"$$v.cl\")" "$(QUIT)" | $(LISP) ; done
+
+
+test: times
+       for v in $(FILES) ; do \
+       echo "(load \"test-help.lsp\")(do-test \"$$v\" \"$(RESULTS)\")${QUIT}" \
+       | $(LISP); \
+       done
+
+clean:
+       rm -f core *.o *.lbin *.bin *.*fasl *~ *.x86f *.fas
+
+times:
+       touch times
+
+tar:
+       tar cvf - *.cl *.lsp makefile  | gzip  -c >  gabriel.tgz
+
+
+
+
+
diff -ru foor/test-help.lsp gabriel/test-help.lsp
--- foor/test-help.lsp  1988-05-03 19:11:31.000000000 +0000
+++ gabriel/test-help.lsp       2004-06-14 20:40:25.000000000 +0000
@@ -6,6 +6,7 @@
   (if (and (not given)
           (setq tem (assoc file *repeats* :test 'equalp)))
       (setq n (second tem)))
+  (setq n (* 400 n))
   (or (eql n 1) (format t "..Using ~a repeats" n))
   (load file)
   (let ((pos (position #\- file))
@@ -15,7 +16,7 @@
        (let ((start    (get-internal-run-time)))
          (dotimes (i n) (funcall command))
          (setq start (- (get-internal-run-time) start))
-         (setq start (/ (float start) n))
+;        (setq start (/ (float start) n))
          (with-open-file
            (st output :direction :output :if-exists :append)
            (format st "~%~:@(~a~)~,12t~,3f"
=============================================================================

2) move the old init.lsp out of the way (to initori.lsp) to use GCL's
   default memory setup

and

3) make LISP=gcl >gcl.out 2>gcl.err ;
   make LISP=lisp QUIT='(quit)' >cmucl.out 2>cmucl.err
   make LISP='clisp -' >clisp.out 2>clisp.err


The clisp results are still running, but it appears clear we will be
faster than these by a considerable margin.  As for GCL vs. CMUCL:

             GCL      CMUCL     CLISP
========================================
BOYER        5.460    20.440    54.510 
BROWSE       13.400   33.330    5.700  
CTAK         3.070    0.830     13.880 
DDERIV       4.310    10.200    12.530 
DERIV        3.700    7.500     43.730 
DESTRU-MOD   4.320    8.710     43.070 
DESTRU       6.810    8.720     14.990 
DIV2         3.730    8.550     104.190
FFT-MOD      0.550    1.750     (in progress)
FFT          0.620    1.320   
FPRINT       4.160    9.150   
FREAD        2.740    4.780   
FRPOLY       18.320   27.990  
PUZZLE-MOD   2.000    29.200  
PUZZLE       2.180    29.130  
STAK         4.950    6.660   
TAK-MOD      2.450    4.290   
TAK          3.050    4.270   
TAKL         0.920    1.590   
TAKR         1.670    2.930   
TPRINT       18.760   19.800  
TRAVERSE     33.400   35.930  
TRIANG-MOD   44.690   531.540 
TRIANG       46.740   96.650  
========================================

Obviously any benchmark can be completely misleading/subject to
selection bias.  Nevertheless, I'd like to include some representative
example of performance in the 2.6.2 release notes providing it is
fair, with adequate disclaimers of course.  So

1) How can this be made more fair (without a lot of work?)?  Are they
   already considered somewhat fair?
2) Are these examples still good representatives in typical lisp
   usage? 

Take care,
-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah