[Gcl-devel] Re: Getting 3-Billion Bytes of Contiguous Memory in a 32-bit Linux

[Camm Maguire]

Greetings, and thanks so much for the recommendation!  Your earlier
email on this subject was also definitely *not* a "dumb idea".

First, one small piece of good news -- we don't need to give libc
malloc any space at all, as GCL redirects it to its own memory
management routines.  So no need to brk everything at the beginning. 

Static linking should definitely be a GCL configure time option.  The
obstacles are all  pretty small.  There are several functions we use
in libc which require dynamic runtime copies of the same version even
when linking statically.  And the plt linking table needs reworking
for this case.  In other words, the compiler can write direct
references to symbols in libc or libm such as cos().  When loading and
linking this code, GCL has a special algorithm to find its address in
the external library, which needs changing in the static case.

I'll put this fairly high up on the queue.  Right now I'm trying to
stabilize some compiler modifications for t4.  Please let me know if
this meets with your scheduling needs.

Also, when done with this compiler work, I'd like to get to si::plet.
This is just a quick note that if you are interested, you can see how
fork works with socket at present under GCL.  We would simply extend
this and make use of the existing 'fasl-vector' read/write routines to
implement a fork-based non-Windows si::plet.

Here is what axiom wants to use to serve up documentation and graphics
to a local web browser: 

(defun bar (s) 
  (let* ((get (read s nil 'eof)) 
         (fn (and (eq get 'get) (string-downcase (read s nil 'eof)))))
    (format s "HTTP/1.1 ~S~%" (if fn 404 500))
    (format s "Content-type: text/html~%~%")
    (format t "get ~a fn ~a~%" get fn)
    (when fn
      (if (probe-file fn)
          (with-open-file (q fn) (si::copy-stream q s))
        (dolist (l (directory fn)) (format s "~a~%" (namestring l)))))
    (close s))
)

(si::socket 8084 :server #'foo :daemon t)

==================================================================
shell prompt> telnet localhost 8084
get /etc/
....

shell prompt> telnet localhost 8084
get /etc/passwd/
....

Take care,



Robert Boyer <address@hidden> writes:

> Hi Camm,
> 
> Warren, with some help from Markus Kuhn in Cambridge, England, where he is
> visiting, has made a quite interesting discovery, one that appears to permit
> the achievement of the Golden Grail of getting 3 billion bytes of contiguous
> memory in a 32-bit Linux.
> 
> A.  First step is go with "static" linking, not dynamic.  No technique has
> yet been found to stop ld from placing some dynamic stuff at 0x40000000,
> thereby breaking the large contiguous chunk somewhere in the middle.  (This
> requires, of course, that --enable-static be made again a viable ./configure
> option and that about .5 megabytes be given over to the space taken by the C
> libraries in the gcl saved image.)
> 
> B.  Before "doing" anything in the executing GCL image (especially anything
> like calling the damned C-library function "printf", which it turns out will
> immediately allocate a buffer at 0x40000000, given a chance, probably because
> of a call to "malloc"), immediately upon program start, do a really greedy
> "brk" to grab your 3 billion bytes.  A subsequent printf will then be forced
> to allocate its buffer elsewhere.
> 
> Performing such an immediate "brk" shouldn't require any physical memory from
> the Linux system until this memory is actually needed.  Separately, with GCL,
> the top of the memory as used by GCL, can be managed in the manner it is
> currently managed.
> 
> Below is a program that shows how to get almost all of the memory (except a
> bit for the stack).  Probably, the "brk" address given is too aggressive, but
> something like 0xb8000000 would work fine and still provide the C-library
> memory allocator a hundred megabytes or so.
> 
> Cheers,
> 
> Warren & Bob
> 
> /*
> 
> test-sbrk.c                                          Boyer & Hunt
> 
> Attempt to raise "brk" to beyond 0x40000000.
> 
> gcc --static -o test-sbrk test-sbrk.c
> ./test-sbrk
> 
> */
> 
> #include <unistd.h>
> #include <sys/mman.h>
> #include <sys/types.h>
> #include <stdio.h>
> 
> main()
> {
>   int brk_int;
>   //  void *setbrk = (void *) 0xbff80000; // brk limit
>   void *setbrk = (void *) 0xb0000000; // brk limit
> 
>   void *bottom;
>   void *top;
>   void *malloc_ptr;
> 
>   bottom  = (void *) sbrk( 0 );
>   brk_int = (int)     brk( setbrk );
>   top     = (void *) sbrk( 0 );
> 
>   if ( (int) bottom == -1 ) {
>     printf("Statement  \"bottom  = sbrk( 0 );\"  Failed\n");
>     return( 0 );
>   }
> 
>   if ( brk_int == -1 ) {
>     printf("Statement  \"  brk_int = (void *)brk( setbrk );\"  Failed\n");
> 
>     printf( "Initial brk address:  %8x \n", bottom );
>     printf( "  Final brk address:  %8x \n", top );
> 
>     return( 0 );
>   }
> 
>   if ( (int) top == -1 ) {
>     printf("Statement  \"top  = sbrk( 0 );\"  Failed\n");
>     return( 0 );
>   }
> 
>   printf( "Large   brk succeeded! \n" );
>   printf( "       Initial brk address:  %8x \n", bottom );
>   printf( "         Final brk address:  %8x \n", top );
>   printf( "Total contiguous memory is:  %u \n",
>         ((unsigned int) top) - ((unsigned int) bottom) );
>   printf( "Total contiguous memory is:  0x%x \n",
>         ((unsigned int) top) - ((unsigned int) bottom) );
> 
>   malloc_ptr = (void *)malloc( 100 * 1024 * 1024 );
>   
>   if ( malloc_ptr == NULL )
>     printf( "Statement \"  malloc_ptr = malloc( 100 * 1024 * 1024 );\" 
> Failed.\n" );
>   else
>     printf( "Statement \"  malloc_ptr = malloc( 100 * 1024 * 1024 );\" 
> Succeeded.\n" );
> 
>   return( 0 );
> }
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah