[Chicken-hackers] [INCOMPLETE PATCH] Heap resizing needs to pick up the pace to match mutation stack resizing! [Was: Re: GC and heap growth percentage question]

[Peter Bex]

On Sun, Jun 24, 2012 at 08:13:56PM +0200, Peter Bex wrote:
> > This would mean either the nursery is bigger than one half of the new
> > space (very unlikely) or that data is copied more than once (which
> > would be a bug).
> 
> I noticed that when the panic happens, gc_mode = GC_REALLOC.  This is
> sort of a special situation.  Could it be the data is copied more than
> once in this case?

I think I (sort of) understand where the problem lies.  When running
the application with -:g and -:d and inspecting it with GDB, you can
see it's done two minor GCs and it's resized the mutation stack just
before it wants to do the major GC that fails.

That major GC runs out of heap while it's looping over the finalizers,
and it decides to longjmp() back and perform a GC_REALLOC, which means
it will create a new heap (twice as big as the current one), and copies
everything to it.

The problem is it's copying the entire old heap *and* copying the full
finalizer list as well as the mutation stack.  Because the test is
creating so incredibly many finalizers, these all get put on the mutation
stack and that stack is resized before the heap is resized (with two minor
GCs happening, so the mutation stack can grow up to twice the nursery size).

When it resizes the heap, it can't keep up and store both the finalizer
pointers (item & proc) from the finalizers list and the whole mutation
stack.

The attached patch is a hack, but it makes the error go away because it
allows the heap size to keep up with the mutation stack size.  It doesn't
take care of the size of the live finalizers, and it mutates heap_size in
such a way that you can't read back the heap_size value to determine the
current heap size; it just hacks the system into thinking that's the
current heap size so when it's growing it needs to take that into account.

I tried something like put these two lines at the top of C_rereclaim2, but
that didn't work; the test still crashed.  Probably because it's not near
accurate enough (not sure why not... probably because I don't *really*
grok what's going on here).

  size += (mutation_stack_top - mutation_stack_bottom);
  size += live_finalizer_count * sizeof(C_word) * 2;


By the way - it seems the finalizer test fails only on amd64 but not
on i386.  (but this was checked with NetBSD/amd64 and Linux/i386, so
it could also be due to other differences)

Cheers,
Peter
-- 
http://sjamaan.ath.cx
--
"The process of preparing programs for a digital computer
 is especially attractive, not only because it can be economically
 and scientifically rewarding, but also because it can be an aesthetic
 experience much like composing poetry or music."
                                                        -- Donald Knuth

heap-size-growth.patch
Description: Text document