[Gcl-devel] Absolut Memory Limit

[Heiko Schroeter]

Hello gcl list,

i'am checking large matrices with gcl (GCL (GNU Common Lisp)  2.6.8 CLtL1    
Dec  1 2007 20:28:22, on Kubuntu x86_64 2.6.24-21-generic, with 2GB RAM) 
and found that i cannot allocate more memory to gcl. The snippet works with 
an array size of 5000 (runtime approx 1.3 seconds).

What i see (or how i interpret it) is that the memory footprint for gcl is 
constantly 262144 pages.
I see a lot of GC for the relocatable pages when the array size is 10000 
(break into debugger after 4 minutes).

Even with various (si:allocate- ) calls when starting the script the total 
amount of 'maximum pages' stays at 262144.
I rechecked this on a Dual Quadcore with 16GB Ram it showes the same amount 
of 'maximum pages'.


What is the recommened way to expand memory for large arrays like this ?

Thanks for any pointer to more infos.
Rgards
Heiko

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Plain gcl -> (room):
>(room)

   434/434    50.7%         CONS BIGNUM RATIO COMPLEX STRUCTURE
     7/45     85.1%         FIXNUM SHORT-FLOAT LONG-FLOAT CHARACTER 
RANDOM-STATE READTABLE SPICE
   155/736    85.9%         SYMBOL STREAM PATHNAME CCLOSURE CLOSURE
     1/2      23.9%         PACKAGE
     2/59     91.8%         ARRAY HASH-TABLE VECTOR BIT-VECTOR
    21/33     98.0%         CFUN CFDATA
    48/59     96.5%         SFUN STRING GFUN VFUN AFUN

   970/2215               1 contiguous (112 blocks)
       26214                hole
       10485   0.1%       1 relocatable

       668 pages for cells
     38337 total pages
    201530 pages available
     22277 pages in heap but not gc'd + pages needed for gc marking
    262144 maximum pages


Array size 5000:
>(room)

 10000/10000   2.1%         CONS BIGNUM RATIO COMPLEX STRUCTURE
 29790/36903   8.9%       8 FIXNUM SHORT-FLOAT LONG-FLOAT CHARACTER 
RANDOM-STATE READTABLE SPICE
  2000/2000    6.6%         SYMBOL STREAM PATHNAME CCLOSURE CLOSURE
     1/2      23.9%         PACKAGE
  1000/1000    0.0%         ARRAY HASH-TABLE VECTOR BIT-VECTOR
  4000/4000    0.5%         CFUN CFDATA
  2000/2000    2.3%         SFUN STRING GFUN VFUN AFUN

   970/2215                 contiguous (141 blocks)
       11210                hole
       56600  86.3%       5 relocatable

     48791 pages for cells
    117571 total pages
     47391 pages available
     97182 pages in heap but not gc'd + pages needed for gc marking
    262144 maximum pages

Array size 10000 (break into debugger after 3min):
>(room)

 10000/10000   2.0%       1 CONS BIGNUM RATIO COMPLEX STRUCTURE
  1000/1000    0.5%       1 FIXNUM SHORT-FLOAT LONG-FLOAT CHARACTER 
RANDOM-STATE READTABLE SPICE
  2000/2000    6.6%         SYMBOL STREAM PATHNAME CCLOSURE CLOSURE
     1/2      23.9%         PACKAGE
  1000/1000    0.0%         ARRAY HASH-TABLE VECTOR BIT-VECTOR
  4000/4000    0.5%         CFUN CFDATA
  2000/2000    2.2%         SFUN STRING GFUN VFUN AFUN

   970/2215                 contiguous (139 blocks)
       40000                hole
       94690   0.0%    5981 relocatable

     20001 pages for cells
    155661 total pages
         1 pages available
    106482 pages in heap but not gc'd + pages needed for gc marking
    262144 maximum pages

#########################################################
; Test snippet

(declaim (optimize (speed 3)(safety 0)))

(progn
  (si::allocate-relocatable-pages 10000 t)
  (si::allocate 'cons 10000 t)
  (si::allocate 'fixnum 1000 t)
  (si::allocate 'symbol 2000 t)
  (si::allocate 'array 1000 t)
  (si::allocate 'string 2000 t)
  (si::allocate 'cfun 4000 t)

  ;; It is not recommended to make hole size less than 1000
  (si::set-hole-size 40000)

  (si::allocate-growth 'cons 1 10000 66 33)
  (si::allocate-growth 'fixnum 1 10000 66 33)
  (si::allocate-growth 'symbol 1 10000 66 33)
  (si::allocate-growth 'package 1 10000 66 33)
  (si::allocate-growth 'array 1 10000 66 33)
  (si::allocate-growth 'string 1 10000 66 33)
  (si::allocate-growth 'cfun 1 10000 66 33)
  (si::allocate-growth 'sfun 1 10000 66 33)
  (si::allocate-growth 'relocatable 1 10000 66 33)
  (si::allocate-growth 'contiguous 1 10000 66 33))

(defparameter arr (make-array '(10000 10000)
                              :element-type 'single-float
                              :initial-element 1.0))

(defun summe-elt (a)
"Just do some silly ops on array"
  (declare (type (simple-array single-float (10000 10000)) a))
  (let ((sum 0.0)
        (wert 0.0)
        (counter 0))
    (declare (type single-float sum wert))
    (dotimes (r 10000)
      (dotimes (c 10000)
        (setf (aref a r c) wert)
        (setf sum (the single-float (+ sum 1.0)))
        (setf wert (the single-float (+ 1.0 wert)))
        (setf counter (the integer (1+ counter)))))
    counter))

(time (format t "My GCL array ops counter: ~A~%" (summe-elt arr)))

############################################################