Re: reentrant C scanner

[John W. Millaway]

Hi Will,

How is the reading going? Any thoughts?  I've made many more changes to flex. 
Some
of them bug fixes, some of the improvements to the API.

1. The main change I'm working now is a nice way to ask flex to generate a
reentrant scanner that fits easily into the "pure" bison parser, since I suspect
that bison is the most common parser used with flex. This will ease the burden 
of
flex users who don't have time to relearn the flex API.

Since bison "pure" parsers expect to call yylex with two or three arguments,
depending upon options, my initial thought is to make a sub-option to option -R
that specifies the number of arguments to yylex. The generated yylex function 
would
be a wrapper that stuffs the arguments supplied by bison into the flex struct, 
then
calls yylex_r.

2. Testing has been been going very well. I have a couple of simple scanners 
and a
multi-threaded driver that pounds on them.

3. The bootstrap configuration in the flex distribution is non-standard, 
however,
I'm no expert when it comes to autotools. In a typical distribution, there is a
bootstrap script (e.g., "bootstrap" or "autogen.sh"), which brings the 
distribution
to the point where it can be built by users. Instead, the flex bootstrap simply
copies "initscan.c" to "scan.c" during the make. This is a small point that I
missed when hacking flex the first time around. I will make a note of it in the
README for developers.

4. What do you think of the macro- heaviness of reentrant flex? It puts a 
burden on
both flex developers and flex users to remember the macros. I wish there 
existed a
better solution (other than scrapping non-reentrant flex or separating the 
codebase
entirely).

The MAIN problem is that non-reentrant flex expects yytext and friends to be
linkable! There is no way around this linkage if flex is to maintain backwards
compatibility with programs that expect to link to yytext. The only viable 
solution
I saw was to macro heck out of it.

-John Millaway


__________________________________________________
Do You Yahoo!?
Yahoo! Auctions - buy the things you want at great prices
http://auctions.yahoo.com/