Re: %option header (Was Re: flex 2.5.13 released)

[John Millaway]

Thanks Bruce. Before I address each issue, let me explain the reasoning behind
the header generation. The header is meant to export only the EXTERNAL API, not
the internal guts. Flex generates many undocumented macros, but that doesn't
mean we're commiting to their existence in future versions. We may rename them
or remove them altogether as the skeleton evolves. The fact that the internal
macros appear in the header file is an undesirable side effect of the header
generation process. In order to undo that side effect, we squash those internal
macros by  #undefing them at the end of the header. This is to ensure that
users don't rely on those macros.

Protecting the namespace is equally as important as protecting the internals.
Flex should not export unprefixed symbols. The start conditions are not
prefixed in the scanner because the were never exported. If a user wants to
export start conditions in the header, then prefixes are a fact of life. As you
said, there is always `sed'.

> 1. yyset_debug is... unconditionally #undef'ed. 
Thanks. I'll fix this.


> 2. start conditions are unconditionally #undef'ed.
The unprefixed ones are undef'd.

> 3. ... It would be nice to be able to avoid the following, none
>     of which are needed (outside of the generated .c file)
Thanks. I look into each one.

> 4. There are a bunch of comments that seem to add little but clutter:
>     various /* %c */, /* %+ */, /* %* */, etc. yy_get_next_buffer,
>     yy_get_previous_state, etc.

These are markers for skeleton hackers, and are invaluable when jumping from
generated code to the flex source to the skeleton and back again. The header is
never going to be remotely as concise and pretty as a human-generated header.

> 5. There are munged versions of start condition names...
>     a. I don't need munged versions; it would be nice to be able to
>        avoid them to prevent namespace clashes
>     b. I do need the real start condtion definitions (currently
>        I extract them from the .c file via sed, but see #2 above).
>        These could be conditionally included by an #ifndef wrapper...
>     c. The munging is inconsistent w.r.t. case (FOOSC_INITIAL vs.
>        foosc_Bar).

They are prefixed in order to prevent namespace clashes. I already have several
scanners that use start conditions such as "string", "comment", "dquote", etc.
They would immediately clash if those names weren't prefiexed. Changing the
case was to adjust to programmer's preference. I've seen flex scanners that use
all UPPERCASE START CONDITIONS, all lowercase start conditions, and First
Letter Capitalization. I tried to make the "munged" symbols follow the
programmer's preference. e.g.,

<DQQUOTE> becomes FOOSC_DQUOTE  (prefix uppercase)
<dquote> becomes foosc_dquote  (prefix lowercase)
<DQuote> becomes foosc_DQuote  (prefix unmodified)
<yydquote> becomes foosc_dquote  (prefix replaced)
<YYDQUOTE> becomes FOOSC_DQUOTE   (prefix replaced)

I'm open to other namespace-friendly suggestions. The "SC_" was added because
people use start conditions like "text", which would be prefixed to become
"yytext" -- clearly undesirable!

> 6. A number of useful macros are unconditionally #undefed...
>     FLEX_SCANNER
>     FLEX_MAJOR_VERSION
>     FLEX_MINOR_VERSION
>     YY_REENTRANT
>     YY_REENTRANT_BISON_PURE
>     yylex

I agree the FLEX_ macros would be useful, but the others are for internal use
only and should not be exported in a header.

> 7. The yylval macro should be wrapped to prevent redefinition
>     warnings (bison defines it if generating the parser with
>     a prefix).

Thanks. I'll fix that.


__________________________________________________
Do You Yahoo!?
HotJobs - Search Thousands of New Jobs
http://www.hotjobs.com