Re: [Chicken-hackers] Regarding #1564: srfi-18: (mutex-unlock) Internal scheduler error

[Jörg F . Wittenberger]

On Dec 3 2018, Peter Bex wrote:

> On Mon, Dec 03, 2018 at 10:46:38AM +0100, Jörg F. Wittenberger wrote:
> > So for me the question remains: wouldn't it be much, much more 
> > efficient to work sort-of hand-in-hand with one of the core developers, 
> > or maybe on the list to get the remaining things (bugs and improvements) 
> > fixed and reviewed?
>
> I think this would be quite helpful.  Perhaps at another hackathon we can
> sit down together (ideally with more than one core developer to ensure we
> all are on the same page and understand it).

Agreed.  Or maybe the list?  Could take time to find a chance to meet.

> This is one of the ugly truths of open source collaborative development;
> you really have to have a good plan on how to communicate the changes
> you're making back to "upstream",[...]
>
> Dropping a complex patch is generally not the way to go about adding code
> to an existing system.[...] eyeball it for obvious mistakes and
> other quality issues.

Too true. Plus: it depends on the culture of the project. Try to put 
yourself in my shoes for a moment.

Up to porting to chicken, I mostly contributed to rscheme, which was a one 
man show. When I hit an issue, I'd send a vague patch and back came a 
completely rewritten one after a day or two. Though it generally where 
small issues or rare failures of complex optimizations going wrong.

> This also means that the submitted code has to be so simple that others
> who aren't familiar with it can study it and debug it if issues crop up
> (and they will, with any sizable change).  The scheduler is a major pain
> point regarding this, since concurrency is difficult enough (or
> impossible?) to understand at all, regardless of the quality of the code
> in the scheduler (which isn't stellar to begin with).

So when I evaluated chicken, I found it to be a compiler producing slightly 
faster code than rscheme.

First tests went well. Then I invested a lot of time until I could run a 
more sizable piece of code.

Just to run into all sorts of issues.

Taking #1564 for an example. It can be quite worse than just killing the 
program: When I ran into it, I was not always so lucky to find the thread 
piled up in the waiting queue to be in a state the consistency check would 
complain about. When the thread was blocked for a different mutex (hence 
sitting in two waiting queues at the same time), the mutex-unlock! would 
happily unblock it - thus stealing the other mutex from the third thread 
holding it.

This kind of poked fun at the idea to use them for synchronization.

At that point it did _not_ occur to me that my code would be especially 
complex a thing. Not did I assume nobody else had ever run more than toy 
examples on load-free systems. I assumed that it was obvious how badly 
broken it was. (And I did not foresee this not coming up elsewhere for a 
decade.)

Sure I felt bad to have to bring up such a huge patch. But it fixed several 
interrelated bugs plus two counts of Big-O reduction of complexity.

I might have expected some questions, comments etc. Certainly not being 
completely ignored.

So I tried to push it for a while.

Same goes for issues, which literally went against the text book examples 
used to teach how not to do things like not using dummy head lists in C - 
something I did not believe anybody would do. At least I expected the 
respective patch to be welcome. Especially as it was quite a job to 
actually change a large file and then test the results before submitting.

Eventually I took being ignored as unwarrented...

...and lost faith in the project.

> So at some point, merging a large change is a bit of an act of faith.
> It also requires trust, which needs to be built up over time by showing
> consistent quality patches and commitment to the project.  This is the
> really hard bit, especially if you just want one specific feature to be
> added and don't have that many other things to contribute to the system
> simply because it works for you.

Yeah, I just needed a compiler as rscheme was dead. I did not want to turn 
into a teacher. Little hope I had that I could ever get something fed back 
upstream. Hence I did no longer try to.

> I don't have a good solution for this, but your suggestion to walk
> through the code together seems like a good one to me.

Agreed.  Chicken is not that bad.  It just has a couple of rough edges.

Best

/Jörg