Re: [Chicken-hackers] Need help to understand C_mutate better.

[Jörg F . Wittenberger]

On Oct 24 2011, Felix wrote:

> From: Jörg F. Wittenberger <address@hidden>
> Subject: Re: [Chicken-hackers] Need help to understand C_mutate better.
> Date: 21 Oct 2011 15:58:17 +0200
>
> > On Oct 21 2011, Jörg F. Wittenberger wrote:
> >
> > > > The mutation stack will
> > > > grow until a garbage collection takes place, so if you
> > > > invoke C_mutate in a C loop without giving GC a chance,
> > > > the mutations will just add up.
> >
> > Reading C_reclaim still without fully understanding how
> > exactly is will interact with C_context_switch
> > it appears to me that if there where always an interrupt
> > pending (or at least too often) then chicken would
> > dispatch to the interrupt handler, which will switch context.
> >
> > I don't see how the C code would return to the (minor) gc.
> > In handle_interrupt is even mentioned that it must not return.
>
> It doesn't - the GC is aborted. Interrupt frequency may simply
> not be too excessive or GC will not run. This means *really*
> excessive, though. Excessively excessive, which might indicate
> a design problem.

What would be the design rational for which interrupt handling
must be able to suppress gc?

Couldn't either the signal handler return to gc, or - my preferred
alternative - the minor gc run prior to the signal handler?
Would make chicken robust to the case at hand.

Also keeping the size of the mutation_stack down in reasonable
bonds would IMHO make chicken better.  (More conservative wrt.
resources.)

When circumstances put too much stress on chicken, then it *can*
run havoc.  If there where no escape, that would be a design issue.

Fortunately escape is: run (gc) in signal handlers.
That way it's transformed into a usability issue.

> > If the next signal arrives before the next gc run this would take
> > the same path.  This way the mutation stack will grow.
> > Slightly at first, but the larger it becomes, the longer it takes
> > to relocate, thereby increasing the probability of yet another turn.
>
> What rate of signals/interrupts would you expect for a heavily loaded
> instance of your application? 

My prog works quite well under normal circumstances, and shows
no problems at all under rscheme.  It then runs in about 70M RAM
and CPU like 0-5%.

There are usually 50-100 subprocesses doing ssl connections.
(Mostly standby) And there is this sqlite3 database.

Once upon a time the database is refreshed from disk content.
Thats a background job.

Upon startup or when the laptop comes back from sleep,
it might find out the database needs refresh.
Chicken will read the disk file by file and talk to the sqlite3 database
(other pthread).

At the same time those 50-100 connection are reattempt.  Several of those
will fail for sure.  Thus we'll see up to retry-times many subprocesses
die off within short time.

This is sometimes enough.  (roughly 50%+/-20%)

> > Given that what I have to do to kickstart the excessive
> > memory consumption it involves massive handling signals and i/o.
>
> You can set "C_post_gc_hook" or "set-gc-report!" to enable debug
> output which will show you if GC's take place. I will also add some
> debug output for mutation-stack reallocation.

Thanks.  -:g -:D plus glib mtrace did the job for me so far.

Also I hope we'll not see it soon again.

(I need remember to remove that (gc) from my signal handler, once I can.)

Best Regards

/Jörg