Re: flex beta 2.5.23 released

[Bruce Lilly]

W. L. Estes wrote:
> On Saturday, 23 November 2002,17:28 -0500, Bruce Lilly wrote:
>
>
> > As distributed, flex 2.5.23 doesn't work w/o the macro (because
> > the appropriate header is not included) or with the macro (because
> > of a syntax error in the path taken by the preprocessor when the
> > macro is defined).  So after all of the contortions, flex still
>
>
> Isn't that last problem a problem with your C preprocessor?

No, it's a problem with scan.c and skel.c as distributed with 2.5.23.
The preprocessor takes the correct branch according to whether or not
the macro is defined.  In one case, the flex code fails to include a
header which is required to define types used by the code (because
scan.c/skel.c does not provide the necessary #include directive); in the
other case, there is a syntax error in a typedef in the scan.c/skel.c code.

> > doesn't build as distributed on some ANSI- and POSIX-compliant
> > systems.
>
>
> Name them. Send us output from the configure/make runs. We'll see what
> we can do. But now, we have no information on what the problem is or
> where it happens, so we can't fix it.

The information *has* been provided and in fact both problems had
already been reported (but not adequately patched). The gcc problem
was reported and correctly identified as C99 non-compliance in:

From: Vincent Caron <address@hidden>
To: address@hidden
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
X-Mailer: Ximian Evolution 1.0.8
Message-Id: <address@hidden>
Mime-Version: 1.0
Sender: address@hidden
Errors-To: address@hidden
X-BeenThere: address@hidden
X-Mailman-Version: 2.0.11
Precedence: bulk
List-Help: <mailto:address@hidden>
List-Post: <mailto:address@hidden>
List-Subscribe: <http://mail.gnu.org/mailman/listinfo/help-flex>,
        <mailto:address@hidden>
List-Id: Users list for Flex,
        the GNU lexical analyser generator <help-flex.gnu.org>
List-Unsubscribe: <http://mail.gnu.org/mailman/listinfo/help-flex>,
        <mailto:address@hidden>
List-Archive: <http://mail.gnu.org/pipermail/help-flex/>
Date: 21 Oct 2002 22:31:04 +0200

and the error in the FLEX_NEEDS... branch was reported in:

From: "Andreas.P.Priesnitz" <address@hidden>
X-Sender: address@hidden
To: address@hidden
Subject: no `long long' type in C++
Message-ID: <address@hidden>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: address@hidden
Errors-To: address@hidden
X-BeenThere: address@hidden
X-Mailman-Version: 2.0.11
Precedence: bulk
List-Help: <mailto:address@hidden>
List-Post: <mailto:address@hidden>
List-Subscribe: <http://mail.gnu.org/mailman/listinfo/help-flex>,
        <mailto:address@hidden>
List-Id: Users list for Flex,
        the GNU lexical analyser generator <help-flex.gnu.org>
List-Unsubscribe: <http://mail.gnu.org/mailman/listinfo/help-flex>,
        <mailto:address@hidden>
List-Archive: <http://mail.gnu.org/pipermail/help-flex/>
Date: Sat, 12 Oct 2002 18:30:15 +0200 (MET DST)

> > The current stable release of flex, 2.5.4a, does not have this problem. It
> > generates .c files which are quite portable.
>
>
> Bruce, that's obvious. That's why the flex 2.5.23 and friends are beta
> releases.

It is appently not obvious to some that the intN_t types in the more recent
beta versions of lex, which are not used in flex 2.5.4a, is the cause of
potability problems.  I have attempted to point that out. Is it still unclear?
If it really is obvious, the the obvious solution is to use the standard basic
types, just as 2.5.4a does.

> We're trying to get flex development done.

Precisely.

> We need to be
> able to add functionality and figure out how to get it working
> everywhere.

Using intN_t instead of the equivalent base C types (as typedef'ed in
the FLEX_NEEDS... branch) does not add functionality.  It *does* lead
to a number of portability issues, which can be avoided by a number of
alternatives, all of which have already been explicitly suggested or
implied:
1. use the basic types, as in 2.5.4a, which are identical to the types
   defined in the FLEX_NEEDS... branch (and which may have the same problems
   on a very few unconventional systems for which short is < 16 bits or
   int < 32 bits).
2. include both the ANSI C99 standard header <inttypes.h> as well as the POSIX
   header <sys/types.h> (the latter to accomodate broken systems based on gcc,
   including Cygwin).  Theoretically, it might be necessary to also include
   <stdint.h>, though that raises additional issues with some systems. This
   option alone won't work for K&R systems and pre-C99 ANSI C systems; for
   such systems it must be compined with option #3 or #6 below.
3. distribute a stub inttypes.h for use with systems that lack that header.
   Maybe also a stub stdint.h.
4. Require per-system configuration for building flex and the .c files
   which it produces.  This is what the latest beta versions using
   FLEX_NEEDS_INTEGRAL_TYPES entails, and it has been described as "a
   very bad idea".
5. require everybody to use the same compiler. Essentially this defeats
   the point of open-source software and platform independence.
6. Wait for all compilers to become fully C99 compliant, to the letter.
   Don't hold your breath, it will be a very long wait.

IMO, option 1 is the cleanest, but it has been rejected to date by the flex
developers...

Since use of the intN_t types instead of the equivalent basic types
a. does not add functionality
and
b. prevents getting flex working everywhere
it seems rather obvious what course of action should be taken, given the
stated goals...

> If you want the stable release of flex, then use it, by
> all means. If you want to use the beta releases of flex, go for it. We
> like having the input on how we're doing, but the price you pay is
> that you are going to have problems.

I can and have used 2.5.4a, including with my own implementation of
reentrant operation via a modified .skel file.  Because I'd like to
see a supported release with reentrant operation, I have been testing
the beta releases and reporting problems.  If you indeed "like having
input on how" those beta releases fail, then there should be no problem
on your side.  On the other hand, if a maintainer of open-source software
doesn't want to hear about problems and/or doesn't want to address the
problems, then that itself is a problem.  On the third hand, if a maintainer
cannot recognize a portability problem as such, then some discussion may
be required.

So...
1. Do you understand that intN_t types are not part of ANSI C prior to C99
   and are not part of K&R C?
2. Do you understand that ANSI C does not mention the POSIX header file
   <sys/types.h>?
3. Do you understand that ANSI C99 requires that if defined, the
   intN_t types be defined in <stdint.h>?
4. Do you understand that the following three C snippets are equivalent
   except for portability and namespace issues?
   A.
       char c;
       short int foo;
       int bar;
   B.
       typedef char frammis;
       typedef short int whatsit;
       typedef int blurfl;
       frammis c;
       whatsit foo;
       blurfl bar;
   C.
       typedef char int8_t;
       typedef short int int16_t;
       typedef int int32_t;
       int8_t c;
       int16_t foo;
       int32_t bar;
5. Do you understand that in the absence of a type definition for intN_t
   types, attempts to use those types will result in errors (indeed, that
   is the case for any undefined type)?
6. Do you understand that a syntax error in a typedef is a fatal error?
7. Do you understand that the following C snippet will not work on fully
   C99 compliant systems because <stdint.h> is not included?
      #include <sys/types.h>
      int16_t foo;
8. Do you understand that the following C snippet will work on fully C99
   compliant systems?
      #include <inttypes.h>
      int16_t foo;
9. Do you understand that the following C snippet will not work?
      typedef char int8_t;
      this line is a syntax error
      int8_t c;
10. Do you understand that the following C snippet will not work on fully
    C99 compliant systems?
      #include <inttypes.h>
      typedef double int32_t;
      int32_t bar;
10a. Do you understand that each line of the snippet above is free of
     syntax errors?
10b. Do you understand that the reason the the snippet in #10 will not
     work is that there is a clash between the explicit typedef and the
     type definition provided via the C99 header <inttypes.h>?

I can provide detailed expplanations of any or all of those issues, but I
have to date assumed that they were understood.

It appears clear that you do understand that per-system configuration
required to use a flex-generated .c file (and that includes scan.c which
is part of the flex distribution) is "a very bad idea".

However, it appears that you might not understand that the use of
FLEX_NEEDS_INTEGRAL_TYPES in fact requires per-system configuration
(manual or automatic) to use flex-generated .c files.  Do you now
in fact understand that, or is a more detailed explanation necessary?

It is clear that you do not understand that the intN_t types are optional
in C99; reference material is provided below (and is also provided in the
earlier posting by Vincent Caron referenced above, which you could have
looked at).

> > > The problem is identifying whether or not the given <inttypes.h> is a
> > > *working* one. It seems that some systems have the file, but the
> > > contents violate the standard.

Please be careful in your attributions; I did not write that. I don't
disagree that a) putting things in flex-generated .c files which requires
per-system configuration is a problem, and that b) some systems with
<inttypes.h> do not fully comply with ANSI C99, but the point here is
that the quoted text is incorrectly attributed.

[...]
> > The *root* problem is that the intN_t types are inherently non-portable:
> > * they are optional, even in C99, which means that there are ANSI-compliant
> >  systems that don't define them
> > * they *may* be defined (in <stdint.h>, included via <inttypes.h> per C99),
> >  in which case there may be a conflict with alternate attempts to
> >  define the same type
> > * there are similar issues with "long long" and "long long" vs. "long long 
> > int",
> >  so even in the absence of a system-supplied type definition, there is no
> >  portable way to get an int64_t as a basic type (as opposed to a 
> >  structure).
> >  [though it's unlikely that any practical lexical analyzer would require
> >  tables so large]
>
>
> and no one will ever need more than 640K RAM.

Non sequitur. I never made any such claim.  I simply observed that there is
no non-portable way in C to get a basic 64-bit type.  You can play ostrich and
pretend to ignore that issue, but it won't go away.  If there are practical
lexical analyzers that require tables larger than 4 billion entries, then
the problem needs to be addressed; if not then the most practical expedient
to getting a portable release (vs. non-portable beta) is to avoid the
64-bit types for the time being.

> Can someone post a confirmation that the int*_t types are optional? If
> that's the case, then we have a very different problem on our
> hands. (Section numbers from the C99 standard or quotations or urls so
> we can all check up on it.)

The reference which I checked and double-checked, then triple-checked
*before* patching and posting was:
"The Dictionary of Standard C", Rex Jaeschke, Prentice-Hall, 2001, ISBN
0-13-090620-4

Quoting page 104, the entry for "intN_t":

"intN_t [C99] A type, defined in stdint.h, that designates a signed integer type
having width N, no padding bits, and a two's complement representation.
For example, int16_t denotes a signed integer type with a width of
exactly 16 bits.

Such types are optional"[...]

and the entry on page 105 for "inttypes.h":

"inttypes.h [C99] A header that includes stdint.h and, for hosted implementa-
tions, extends it with functions relating to greatest-width integers."[...]