Re: [Chicken-hackers] PATCH: more efficient scheduler

[Jörg F . Wittenberger]

Am 18.05.2016 um 21:17 schrieb Peter Bex:
> On Mon, Feb 15, 2016 at 01:57:52PM +0100, Jörg F. Wittenberger wrote:
>> Hi,
>>
>> the attached patch does not fix any bugs.  It brings some performance
>> improvements under certain load conditions.  The cost: It adds the
>> overhead of another counter and a dispatch on a a fixnum modulo
>> operation - IMHO negotiable.
>>
>> The change is helpful when sits watching on several file descriptors for
>> requests _and_ uses several threads, message passing etc. to produce
>> responses.
> 
> Hi Joerg,
> 
> I've tested with some varying types of load with Spiffy (using wrk, ab
> and siege).  The results are inconclusive either way.  The patch doesn't
> seem to make much of a difference, at least with a reasonably simple
> Spiffy setup.
> 
> This is quite an invasive patch, which moves whole swaths of code and
> also complicates the scheduler further by adding more lists and tweaks,

I can understand.  What I'd like most would be a way to pull the
scheduler out of the core system completely.

The dual use and interference with force-primordial adds a lot of
complexity to the current implementation as your related finding in the
other thread shows.

Use cases have different load or even just users want to play with
scheduling policies for the fun of it.

I can't proof the implementation I ran for the past years to work under
every kind of load not yet tried.  But it is IMHO simpler.  Cuts out
force-primordial, does not need to sys#dispatch-interupts which in turn
could be removed everywhere.  Nobody is going to swallow it as a whole,
but in piecemeal supporting those it -sure- increases code complexity.

> and even removes some things like the check for C_signal_interrupted_p,
> which makes me very hesitant to just apply it for an alleged performance
> boost.  On the whole, I think it's too dangerous to fiddle with the
> scheduler unless it brings either considerable simplification of the
> scheduler's code, or measurable improvements for common workloads.

For varying values of "common workload" ;-)

In my case the standby load is about 50 child processes and 100-200 open
file handles.  Most of the latter are non-critical as they connect to
network but the one signaling callbacks from external pthreads is
critical.  One there is load there can be several hundred short lived
chicken threads passing messages around.

> Also, you did mention getting segfaults with your version, so who's
> to say this change doesn't *introduce* any segfault crashes?

The code was running that way for about eight years or so.  Segfaults
came and went in this time.

At the moment I don't recall when precisely, but I did not see those
segfaults maybe mid-March.

> So, unless you can show us how to reproduce this measured improvement,
> I think it's best not to apply this patch.

I see, the piecemeal approach will not work.  We should work towards an
API for the scheduler to hook into and remove it from core.

Cheers

/Jörg