Re: [Gcl-devel] Recent HEAD changes

[Camm Maguire]

Greetings!

"Mike Thomas" <address@hidden> writes:

> Hi Camm.
> 
> I've been keen to try out your new optimisations to GCL.  I'm running
> through those changes you've been making over the past few days and thought
> you might like to hear some comments from the Windows perspective.
> 

Yes, indeed, thanks!

> HEAD doesn't configure or run on Windows as a result of those changes.  I've
> made some local changes to help this situation but have some questions in
> addition to those comments all of which are set out below.
> 
> 1. Your newly checked in test for the TYPE_BITS macro causes a fatal end to
> the configuration script.
> 

??? Need more detail here -- what is the problem?

> 2. I'm getting a negative CSTACK_ADDRESS -2147483648
> 

This is normal if accurate.

> 3. Are these sensible values?
> 
>   #define CSTACK_TOP 0x80000000
>   #define CSTACK_RANGE 0x40000000
> 

Yes, It appears as if your C stack counts down from 0x80000000,
leaving the range above this value fair game for immediate fixnums. 

> 4. What are these macros meant to represent?
> 
>   SHARED_LIB_HEAP_CEILING

The upper bounds on heap/core addresses before we run into shared
memory load addresses

>   IM_FIX_BASE 0x80000000

The beginning of the immediate fixnum table. 

>   IM_FIX_LIM 0x40000000

The size of the imm fix table, half the top range as we need a high
bit for marking.

> 
>   (Perhaps some comments in "h/gclincl.h" would help?)

Yes!  Could you suggest some?

>   There is no "ldd" on Windows so the shared lib heap ceiling test is
> meaningless.   DLLs are loaded above the main program text etc and below the
> heap.

How can one discover this programmatically?

In short, during my visit to UT, it has become apparentl that there
are many 'serious' GCL users, by which I mean industrial users pushing
the limits of GCL on their machines, pushing to n-way 64bit GCL with
gigs of ram, etc.  I've tried to redesign the memory configuration
process to make the most of the address space available.  There is
doubtless more that needs doing here, but the outline is as follows:

DBEGIN is ideally a compile time determined constant which is as low
as possible.  On Linux, we lower it if we can via an edited linker
script.  In most cases, we start at 0.  In actuality, of course, there
is code text and static data taking up the first little bit of this
area, with the actual heap beginning somewhat higher, but not much in
the grand scheme of things.  Page types are tracked from DBEGIN up to
DBEGIN + MAXPAGE*PAGESIZE.

Eliminate any unnecessary restrictions on heap growth.  Eventually, I
would like to make type_map and sgc_type_map dynamic arrays in the
heap, making the only effect of MAXPAGE being to protect the heap from
uncontrollable obstacles elsewhere in the address space.  For example,
without kernel modification, shared libraries start loading on 32bit
linux at 0x40000000, putting an absolute cap on the heap of 1Gb.  It
could be argued that it would be better to start above this value
where one could get almost 2Gb before running into the stack which
counts down from 0xc0000000, but I'm banking on someone figuring out
how redhat/oracle has hacked the kernel to move the shared libraries
just beneath the stack and counting down, leaving an undivided user
memory space up to 2.7Gb on 32bit Linux.  For the purposes of
clarification, GCL heap counts up from approximately DBEGIN via sbrk
(or emulated), with no mmap, and malloc redirected to sbrk, so the
only non-sbrk memory in the image is that mmaped by the system's
dynamic linker loader at image start.  Therefore the only two hard
limits to the heap are the shared memory load start, and/or the C
stack lower bound.  Ideally we can separate all these regions of
memory effectively so we have no bizarre collisions.  All these
considerations will further restrict MAXPAGE if necessary.

Given DBEGIN and MAXPAGE, try to grab as much C stack as possible to
avoid artificial limits on recursion depths.  Bound the C stack if it
counts down by the max heap (and ideally if I can figure out how to do
it, by the upper bound of the shared library memory, though this would
have to be done at runtime, as we can change this via compiler::link).
If the C stack counts up from high addresses, bound it to half its
range to make room for an immediate fixnum table unless disabled.
>From this, get a max used mem address, which I'm incorrectly calling
CSTACK_TOP for the moment, though it need not be this if the C stack
is below DBEGIN.  I assume that the C stack start and direction are
system immutables.

Given this, use the remaining high addresses, which are inaccesbile to
the heap under any circumstances, for immediate fixnums.

Getting this all to work portably will take a little sweat, but I
don't think too much.  We're close to autoconfiguring all of this
across the Debian 12.

> 
> 5. On Windows, the C stack lives below the program text, data and heap and
> its size defaults to CSSIZE 1073741824 but can currently be altered by
> configure to some other value as required.

OK, as you might have notices, everyone can configure CSSIZE now too.
Your defaults might no longer be needed, but in any case would be
great if you could give it a look over to streamline things to the
extent possible.

We also have sigaltstack based continuation on c stack overflow in
main.c.  Might want to check how this would work on mingw too.


> 
> 6. The Lisp heap start is normally hard-wired by our configuration script to
> start at DBEGIN =  0x1a000000.  I am thinking that I could lower that start
> point or just remove it altogether - the heap start address can be altered
> automatically as the system tries to grab heap memory anyway according to
> whether it succeeds in getting the blocks it asks for.  How this would
> affect unexec I am uncertain at this stage.
> 

My understanding is that DBEGIN should never move after compile time.
I would suggest setting it as low as possible, and ideally, having
this determined from configure magic to flow with future changes to
the mingw address space layout.   It sounds as if all your obstacles
are below DBEGIN anyway, meaning you can likely have the largest GCL
heap of all the ports!  But I don't understand your statement that the
C stack is below the text if you are getting a negative
CSTACK_ADDRESS. 

Take care,

> Cheers
> 
> Mike Thomas.
> 
> 
> 
> 
> _______________________________________________
> Gcl-devel mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/gcl-devel
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah