Re: [Chicken-hackers] CHICKEN in production

[Peter Bex]

On Tue, Oct 07, 2014 at 01:13:09AM +0400, Oleg Kolosov wrote:

Hello Oleg,

Thanks for providing some more information about your project!
I think these kinds of postmortem analyses are very interesting
and we should take to heart all the lessons learned, and use them
to improve CHICKEN.

> We are trying to avoid using Chicken as a ‘glue’ because we figured that FFI 
> transitions can be major bottleneck (especially strings).

The overhead of calling C should be pretty minimal in the usual cases,
unless strings are the only problem.  If it's the only dealbreaker,
I think that should be fixable.

> And adding Chicken to a C program makes normal analysis and debugging tools 
> pretty much useless (for finding memory leaks and such), so hardware 
> interfacing layer is pure C with separate high level FFI bindings on top.

It takes some more practice, but debugging C code called from CHICKEN is
quite doable in my experience, but then I've never done huge C & CHICKEN
projects, only smaller libraries.  Could you explain a bit more what the
problems are you ran into?

> We also struggled with posix and process control functions a lot (long 
> story), trying to be functional here backfires badly, so we ended up with 
> straightforward and ugly code (looking like verbose C with parentheses), 
> replacing some functions from standard library (namely process-run) and 
> customized error handling.

Would you care to unpack this a little?

> There was a few problems (I don’t remember clearly) with preemptive 
> scheduling, so we are using strategically placed carefully adjusted sleeps 
> with manual yields. I’ve borrowed a few ideas from Chicken implementation and 
> made a video player (used for background: pure C, no FFI, no GUI) abusing 
> libuv event loop for CPS trampoline. The code looks strange for casual 
> observer but performs surprisingly well. I’ve not yet figured out how to wrap 
> this for an egg (managing C callbacks is hard).

Sounds interesting.  So at least you got something out of it aside from
just frustration ;)

> So, in the end, there are some great things (see video in this thread) to 
> showcase, but for me (low-level and performance stuff mostly) it was more 
> pain than joy.

If you can pinpoint the exact places where performance is particularly
bad we can (at least attempt to) fix them.

> There are hot internal discussions currently about migrating to something 
> more widely supported (with proper debugger, profiler, and other useful 
> tools) for our next big project, because a new hardware is more powerful and 
> there are fewer restrictions.

This is a bit of a tidal function: sure, hardware gets faster every year,
but then they invent some new class of device which is more constrained
than the previous generation, or a new niche of computation evolves where
every CPU cycle is precious (bitcoin mining? 3D games?).  So even though
there are lots of people falling over eachother trying to tell you that
"hardware doesn't matter" and you should use their slow-ass language,
that's just bullshit: performance will *always* matter.

Cheers,
Peter
-- 
http://www.more-magic.net