gcl-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Gcl-devel] Re: Boyer benchmark results


From: Eric Marsden
Subject: [Gcl-devel] Re: Boyer benchmark results
Date: Mon, 28 Jun 2004 15:53:54 +0200
User-agent: Gnus/5.1006 (Gnus v5.10.6) Emacs/21.3 (gnu/linux)

I apologize for coming in late in this discussion, but I don't read
this list very regularly.

I am the maintainer of the cl-bench suite that has been discussed in
this thread. This aims to provide reasonably objective performance
comparisons between ANSI-compliant CL implementations, but to a
certain extent accepts workarounds for ANSI compliance bugs in
different implementations. Until recently GCL had a serious bug in
EVAL-WHEN handling that prevented cl-bench from running, but Camm just
fixed it (thanks!), inspiring me to retry the suite with GCL.

A few bug reports later, I have obtained the following provisional
results. These are run on a dual-processor 1.1GHz i686 machine running
GNU/Linux; the same optimization qualities are used for all
implementations: (speed 3) (safety 0). They were not carried out in
proper (zero load) benchmark conditions, so please do not quote them.

,----
| Benchmark                 Reference     GCL    SBCL    CLISP   
| ------------------------------------------------------------
| COMPILER                 [      3.70]  -1.00   1.77    0.77   
| LOAD-FASL                [      0.36]  -1.00   2.22    2.22   
| SUM-PERMUTATIONS         [      4.30]   1.18   1.55    1.59   
| WALK-LIST/SEQ            [      0.19]   1.89   0.37    0.53   
| WALK-LIST/MESS           [      0.19]  13.68   0.32    1.84   
| BOYER                    [      8.73]   0.18   1.13   11.42   
| BROWSE                   [      0.81]   4.44   1.44    2.69   
| DDERIV                   [      0.53]   3.89   1.77   12.49   
| DERIV                    [      0.61]   3.59   1.82   10.93   
| DESTRUCTIVE              [      0.82]   3.49   1.23    4.05   
| DIV2-TEST-1              [      1.07]   3.87   1.67    8.92   
| DIV2-TEST-2              [      1.54]   2.96   1.45    7.09   
| FFT                      [      0.14]  30.21   0.79   52.36   
| FRPOLY/FIXNUM            [      0.83]   2.41   1.18    9.22   
| FRPOLY/BIGNUM            [      1.09]   0.83   1.06    4.70   
| FRPOLY/FLOAT             [      1.29]   1.40   1.21    7.97   
| PUZZLE                   [      0.51]   5.43   1.25   34.41   
| TAK                      [      0.71]   1.55   0.94   10.89   
| CTAK                     [      0.71]   2.65   1.11    7.34   
| TRTAK                    [      0.67]   1.64   1.06   11.45   
| TAKL                     [      1.18]   0.68   0.75    9.64   
| STAK                     [      0.85]   1.02   1.69    7.64   
| FPRINT/UGLY              [      1.46]   1.23   1.75    1.47   
| FPRINT/PRETTY            [      5.18]   0.34   1.34    1.23   
| TRAVERSE                 [      1.51]   3.16   2.15   12.52   
| TRIANGLE                 [      1.30]   2.35   1.12   25.59   
| RICHARDS                 [      0.95]   1.24   1.24   23.87   
| FACTORIAL                [      0.68]   1.69   1.24    6.84   
| FIB                      [      0.82]   4.41   1.01    5.40   
| FIB-RATIO                [      0.66]   9.59   1.08    0.09   
| ACKERMANN                [     10.38]  17.32   1.04   11.07   
| MANDELBROT/COMPLEX       [      1.44]   1.44   1.26    4.62   
| MANDELBROT/DFLOAT        [      0.12]  10.08   0.50   38.42   
| MRG32K3A                 [      1.28]   4.39   1.01   64.30   
| CRC40                    [     33.12]   7.14   1.28    7.70   
| BIGNUM/ELEM-100-1000     [      0.97]   0.72   1.05    0.10   
| BIGNUM/ELEM-1000-100     [      4.95]   0.40   1.01    0.05   
| BIGNUM/ELEM-10000-1      [      5.42]   0.32   0.97    0.04   
| BIGNUM/PARI-100-10       [      1.19]   0.03   0.98    0.03   
| BIGNUM/PARI-200-5        [     14.19]   0.01   0.96    0.01   
| PI-DECIMAL/SMALL         [     47.36]   4.72   0.97    0.09   
| PI-DECIMAL/BIG           [     95.08]   3.50   0.96    0.02   
| PI-ATAN                  [      2.42]   3.75   1.14    3.06   
| PI-RATIOS                [      8.92]   2.41   1.02    0.12   
| HASH-STRINGS             [      0.77] 165.82   1.04    3.09   
| HASH-INTEGERS            [      1.66]   0.82   1.63    2.55   
| SLURP-LINES              [      5.10]   1.12   1.07    1.65   
| BOEHM-GC                 [      2.52]   3.38   1.62   14.95   
| DEFLATE-FILE             [      1.29]   1.74   0.97    5.40   
| 1D-ARRAYS                [      0.20]   0.00   1.35    7.75   
| 2D-ARRAYS                [      1.77]   0.05   0.51   21.36   
| 3D-ARRAYS                [      4.83]   0.04   0.60   16.59   
| BITVECTORS               [      1.74]   0.01   1.32   13.77   
| BENCH-STRINGS            [      4.60]   0.00   0.24    0.72   
| fill-strings/adjustable  [     47.71]   0.00   0.94    2.02   
| STRING-CONCAT            [    146.96]   0.00   0.42    6.59   
| SEARCH-SEQUENCE          [      4.94]   0.00   0.11    3.75   
| CLOS/defclass            [      6.55]   0.23   0.73    0.11   
| CLOS/defmethod           [     12.99]   0.24   1.07    0.01   
| CLOS/instantiate         [     18.00]   0.31   0.95    0.64   
| CLOS/simple-instantiate  [      0.45]  98.27   1.33   13.24   
| CLOS/methodcalls         [      2.89]   1.83   1.66    5.25   
| CLOS/method+after        [      9.74]   1.15   1.48    0.67   
| CLOS/complex-methods     [      0.14]   8.00   7.50   -1.00   
| EQL-SPECIALIZED-FIB      [      0.67]   2.45   1.06    4.28  
| Reference time in first column is in seconds; other columns are relative
| Reference implementation: CMU Common Lisp 18e+, level-1 built 2004-06-11 on 
melbourne
| Impl GCL: Kyoto Common Lisp GCL 2.7.0
| Impl SBCL : SBCL 0.8.10.19
| Impl CLISP: CLISP 2.32 (2003-12-29) (built 3281883514) (memory 3281883691)
| === Test machine ===
|    Machine-type: X86
|    Machine-version: X86
| Linux melbourne 2.4.17 #2 SMP Wed Feb 6 11:35:03 CET 2002 i686 GNU/Linux
`----

Note that the numbers in all columns except for the first are
relative, so for instance 1.18 means that GCL 2.7.0 with --enable-ansi
from anoncvs is 18% slower on the SUM-PERMUATIONS test than
CMUCL-19a-pre. A value of -1 means that the test was not run for that
implementation (probably due to a bug in that implementation).

It is interesting to note that GCL is a lot faster than CMUCL on
certain tests such as BOYER, the array tests and the bignum tests; I
suspect that GCC/GCL is "cheating" on some of the array tests by
deleting unused code. Note however that it is hugely slower than CMUCL
on many other tests, such as FFT and ACKERMANN. 

I strongly suggest that people interested in performance comparisons
use cl-bench as a basis for their measurements, for a number of
reasons:

   - it includes all the Gabriel benchmarks, but with a few bugfixes
     and with some rewriting to more closely ressemble typical CL
     programming style.

   - I have attempted to include more modern CL benchmarks, such as
     the CLOS code, the bignum processing code, array and hashtable
     tests etc. These should give greater coverage of a CL
     implementation's performance characteristics.

   - the benchmark methodology attempts to be as fair as possible
     between implementations; for example I consider Camm's prelinking
     FTYPE declarations for GCL to be cheating. Other factors to
     consider are running implementations with the same heap size, and
     disabling "development-only" settings such as collecting XREF
     information.

The version of cl-bench that is currently on my web site does not run
in GCL, but I shall shortly make a new release that does work. 

-- 
Eric Marsden                          <URL:http://www.laas.fr/~emarsden/>





reply via email to

[Prev in Thread] Current Thread [Next in Thread]