It is really fun to see so much activity the last days. Thanks!
I think I solved my problem that moment by calling.
get_tok_str(tok, &tokc));
versus
get_tok_str(tok, NULL);
After a bit of experience with the code, I could even accept that
the original behavior of the function is just fine.
What is missing is few lines more of comments. As with any code base once
you know it any piece of documentation becomes superfluous, but for
newcomers
the amount of comments will clearly decide the "should I stay or should
I go" ;)