Re: [Chicken-hackers] CHICKEN in production

[Oleg Kolosov]

On Oct 8, 2014, at 11:16 AM, Peter Bex <address@hidden> wrote:
> 
> On Wed, Oct 08, 2014 at 01:31:30AM +0400, Oleg Kolosov wrote:
>> On Oct 7, 2014, at 10:04 PM, Peter Bex <address@hidden> wrote:
>>> The overhead of calling C should be pretty minimal in the usual cases,
>>> unless strings are the only problem.  If it's the only dealbreaker,
>>> I think that should be fixable.
> 
> The upshot of this is that a string will need to be checked at most
> once, and never copied (except as is done normally, by the GC).
> This will need some looking into, because I'm not sure we have any
> bits to spare in the representation :S

Well, at least it is fixable. We will be happy to test this on MIPS and ARM 
platforms. Take a note that both of them are 32bit, so “spare bits” might 
become real problem.

Side note. What do you think about https://github.com/antirez/sds? It’s just 
single C file - easily embeddable. We are using it since the dawn of time 
without problems. Maybe studying or even merging it would help with the cleanup.

> 
>> Additionally, there was some performance problems with unicode handling. So, 
>> now we use libc locale functions for conversions and doing indexing and 
>> processing in C. This is pain but at least 3 times faster than Chicken. 
>> There are still a lot of trickery on the GUI side to provide responsive 
>> incremental search because the amount of data returned is still quite large. 
> 
> Unicode is still tentatively on the TODO-list for CHICKEN 5.  If you
> have any useful tips or suggestions, that would be helpful.  But to be
> honest I'm not sure we'll get around to it: there are still a lot of
> pending patches on this list.  I'm not through polishing the numbers egg
> yet, and there hasn't been any visible effort at making the core modular
> aside from the "modular compiler" patch.  So much to do, so little
> manpower :(

Unfortunately I do not have much experience with encodings. I’m pushing bits 
around (drivers and such) most of the time. Alex Shinn seems to know this stuff 
well. There was a topic about this on chicken-users some time ago. Let’s do 
what he suggests: cursor API and specialized map/fold for utf-8 strings instead 
of srfi-13.

Actually, compared to few other solutions, Chicken’s utf-8 support is doing 
pretty well, considering that it’s written in Scheme. It is just our use case 
that is so performance sensitive.

> Yeah, that can certainly be a problem.  Debugging CHICKEN code can be a
> bit like reading tealeaves, sometimes.  I don't see an easy way to fix
> that, considering Cheney on the MTA is really an essential part of CHICKEN.

We believe that this is fixable. Cheney is an obstacle of course, but can be 
worked around with some help from the C backend. We had limited success with 
this - there are various corner cases and not much time. We hope that after 
some internal cleaning (CHICKEN 5 and beyond) the task will become more 
approachable.

> 
>> And useful info about passed arguments and such is left in the generated 
>> comments - you need to inspect the sources with the ‘list’ command to view 
>> it.
> 
> That's a good thing, right?  The sources are available and can be kept
> using the -k switch.  If there are other comments that could be inserted
> which might be more helpful, we're open to suggestions.

Yes, this is what we are doing. If only there was a way to somehow give this 
info to the debugger. Maybe with some extension or plugin. This needs more 
investigation.

> 
>> We tried to improve this with the insertion of #line directives without much 
>> success - code generator is too complex, especially where FFI is involved. 
>> We are inserting logging statements everywhere. Unfortunately logging 
>> considerably uglifies the code and makes some functional programming idioms 
>> much harder to use (like map/fold/cut oneliners). Also various analysis 
>> tools like Valgrind and libc malloc checkers fall flat when Chicken is 
>> involved.
> 
> I think Jerry mentioned he used Valgrind for debugging CHICKEN/C code,
> and malloc checkers should work just fine with CHICKEN: it doesn't mess
> with the C heap.

Maybe I’ve missed something, but everything I’ve tried gave me endless lists of 
leaked memory pointers. Maybe this can be amended with some special 
configuration or plugin. Building modern analysis tools is quite hard with our 
ancient toolchain, so I’m stuck with whatever the upstream supports - those are 
usually tightly coupled with the device SDK. Having specialized memory checker 
with CHICKEN knowledge could be really helpful.

>> We are trying to simulate parallel processing and separate responsibilities 
>> with the worker processes communicating through sockets. There are also 
>> message passing threads involved for monitoring and control. Judging by the 
>> history this may be the most buggy part of the project. With numerous 
>> workarounds and special case handling. SIGINT handling is still buggy, but 
>> not critical for production. Yes, the task is complex, but the API is too 
>> confusing and fragile too. It might be adequate for C but in Scheme a lot of 
>> foots was shoot away.
> 
> Perhaps the new "hardwood" egg could be used once it grows the ability
> to spawn nodes.  There's also concurrent-native-callbacks.  But yeah,
> this is a part where tools are lacking.

The egg API looks quite promising, but CHICKEN’s own process handling code 
needs cleanup too. The main problem IMO is too much mixed inline C and Scheme 
code. I think that even rewriting it completely in C would shrink the 
implementation quite a bit. I do not propose doing that - just illustrating the 
point.

> What exactly is wrong with SIGINT handling?

I have no idea. This remark was about the part of our project which was really 
messy. After adding live coding with hot reload it has become “abandon all 
hope…” kind of messy. We’ve discussed the ways to get rid of it just the other 
day and Erlang was mentioned. We will investigate if it is possible to use the 
hardwood egg here. 

> 
>>> Sounds interesting.  So at least you got something out of it aside from
>>> just frustration ;)
>> 
>> There was some discussions about replacing Chicken scheduler with libuv 
>> event loop and providing filesystem and socket API on top of it. The 
>> scheduler modification is necessary to block green threads to simulate 
>> synchronous calls. There are a lot of custom and confusing code in Chicken 
>> around select function with workarounds for Windows. We think that libuv 
>> implementation is superior. There are some concept code but we’ve not 
>> progressed too far with this yet.
> 
> This has been suggested before.  (how) does libuv really fix this
> situation?

Just from the point of view of replacing custom code with heavily tested and 
externally supported library to ease the maintenance burden. Their Windows and 
OS X support seems much better (and epoll is faster on Linux too, AFAIK), and 
the code is quite readable (IMO). The API could help with the refactoring the 
posix unit. Some big projects are using it. CHICKEN could have some specifics 
which makes the conversion impossible, but from the first sight it seemed 
pretty doable.

>> Scheduler even with disable-interrupts is still active - very hard to 
>> diagnose, but mysterious bugs are fixed by going down to C and not returning 
>> back until everything is settled (like fork -> exec). It would be nice to 
>> have an option to get rid of it, i.e. for performance critical parts we 
>> would like to have complete manual control - without interrupt handling and 
>> such code inserted.
> 
> Have you tried calling C_disable_interrupts()/C_enable_interrupts()
> from C?  This should give you complete control of the scheduler.  If you
> (declare (disable-interrupts)), interrupt checks will not be inserted
> in the generated C code of that compilation unit.  However, if you
> call any external code which was compiled without that declaration,
> of course interrupts will still be checked while that code is running.

I could not disable interrupts everywhere because it would affect other code. I 
was in a hurry, not diagnosed the problem too much, and just rewrote the 
offending part in C. There was no problems with it since then.

> I'd like to note, though, that it's unfortunate that we're only now
> having this conversation about your previous project: I'm sure that if
> you'd asked earlier on the list, we could've helped you better to debug
> or work around problems.  Like for example the disable-interrupts and
> defstruct issues could possibly have been solved as you ran into them.
> One of the great things about CHICKEN is its community; I'd advise
> everyone to make good use of it!

It seems that I’ve given the wrong impression in this thread. I’m writing about 
the problems because it is what concerns me the most - it’s my job. No 
technology is perfect. And I like to complain to stir some discussion. It’s not 
CHICKEN specific, sorry about that.

From the technical standpoint I could say that the project is a big success. 
And it is largely thanks to CHICKEN. When we started about 2 years ago - nobody 
was proficient in Scheme or functional programming for that matter. There was 
some ideas and prototypes only. Using unknown dynamic language for memory and 
CPU constrained platform was a huge risk. But we decided to play a wildcard 
because there was a previous attempt by another team using more traditional 
technology (C++) which essentially stalled by feature creep. Our competitor 
released similar device recently. They have much more powerful hardware and we 
are still doing better in some areas which I’ve mentioned as problematic: 
smooth drawing (scheduler) and responsive search UI (string passing). It is 
because we spent so much effort tuning these.

And going to production is not the end - it’s the beginning. Our product is 
long term support. It’s expected that we will maintain the platform for about 5 
years to come. There are still many not implemented features. Hopefully, there 
will be more time to actually figure out longstanding problems instead of 
writing code quickly. So, we still want all the improvements discussed here and 
planned for CHICKEN 5.

The discussions are about the new platform which is quite ambitious and still 
has a long way to go. We actually would like to avoid rewriting or maintaining 
separate versions of the software.

— 
Regards, Oleg
Art-System