Re: Some issues wrt compilation on Windows using VS.NET

[Tim Van Holder]

> On Thursday, 10 April 2003,16:17 +0200, Tim Van Holder wrote:
> 
> > Steps taken so far:
> > 
> > - created a win32/msvc.net subdir (please advise as to
> >   what a preferred location would be for portability
> >   cruft like this)
> 
> let's say:
> 
> portability/platform-name
> 
> where "platform-name" is something "reasonable".

OK - so portability/msvc.net?

> > - manually created a config.h from config.in and put in
> >   win32/msvc.net
> 
> You can't run configure on this platform? configure only requires a
> born style shell to be executed. Does this platform lack one for some
> reason? If so, then this platform is most likely not POSIX-compliant
> and you're going to run into lots of issues with flex.

There are bourne shells available for windows, but none are part of a
normal Visual Studio setup, which is what I was aiming at.
This is going to proide, well, challenges, if I am to recreate the
flex.skl -> skel.c transformation without using any non-standard tools
(if that is at all possible).
The standard practice in most GNU packages I've seen so far is to have
a config.h.msvc or similar, premade for a non-unixy Win32 platform, so
I did the same here.
Also, the tarball from sourcefourge didn't actually contain a configure
script, and while I have a few cygwin tools installed for convenience,
I did not want to get autoconf, automake and their dependencies just for
the sake of making a config.h that was easy enough to make by hand.

> > - added GNU Regex (regex.[hc]) to win32/msvc.net
> 
> This is probably best handled by including a lib/ subdirectory and
> giving configure the appropriate tools to make decisions 
> about what to use.

True.  But as this target will not configure I didn't make any such
modifications.  Or you could compile regex.c unconditionally into the
library.  CVS took this approach I believe; they only have this in
configure.in

dnl
dnl Force lib/regex.c to use malloc instead of messing around with
alloca
dnl and define the old re_comp routines that we use.
dnl
AC_DEFINE(REGEX_MALLOC, 1,
[Define to force lib/regex.c to use malloc instead of alloca.])
AC_DEFINE(_REGEX_RE_COMP, 1,
[Define to force lib/regex.c to define re_comp et al.])

> > - created solution and vc++ win32 console project; added
> >   all sources & headers, using default settings
> 
> This platform doesn't have a make program? or a program which can read
> Makefiles? (I know that there are possibly some issues reading GNU
> Makefiles, but it sounds like you've got no make at all. Is 
> this right?)

Visual Studio _does_ have a (non-standard) make program, but that
is not the canonical way to build things on this platform, so I made
the typical project files used by the IDE instead.

VS.NET no longer has VC++ 6's ability to export a 'real' Makefile from
a project; otherwise it would have been easy to include one.  But a
makefile is no longer needed for VS.NET, as the IDE's project files can
be used from a console as well:

   devenv flex.sln /build [Debug|Release]

builds the program in the debug or release configuration as required;

   devenv flex.sln /clean [Debug|Release]

cleans up after itself.


> > - added scan.l, parser.y, and mkskel.sh (+ flex.skl) to
> >   the project as well, in preparation for custom build
> >   rules.
> 
> erm, you made separate copies of scan.l etc.? I'm unclear what you did
> in this step (and probably why you did it as well.)

Nono, I added them to the IDE's project, so I could (at a later time)
define custom build rules to build scan.c, parser.c and skel.c
from them.  Currently, the IDE will ignore these files and only look
at the .c and .h files.

> > - (minor) size_t vs. flex_uint32_t
> >   -> VS.NET is a 64-bit ready environment, and by
> >   default warns about 64-bit portability issues.
> >   size_t is marked as potentially 64-bit in size; this
> >   makes it unfortunate that size_t values are assigned
> >   to flex_uint32_t fields (in table structures).  Since
> >   those fields are in fact sizes, it would make more
> >   sense to have them as size_t instead of flex_uint32_t.
> >   This only causes a bunch of (easily disabled) warnings
> >   though, so it's not that big a deal.
> 
> We've had a lot of trouble re integer portability. At some point we
> can do an audit and resolve things one way or another. Thanks for
> pointing this out.
> 
> > - (minor) signed vs unsigned
> >   -> newsz in yytbl_data_compress() should be a size_t
> >   (or at the very least a flex_uint32_t) instead of a
> >   flex_int32_t.
> 
> noted. and see previous note as well.
> 
> > - yytbl_write32() uses htonl() unconditionally and
> >   yytbl_write16() uses htons() unconditionally.
> >   The netinet/in.h header is checked for but the functions
> >   are used regardless.
> >   -> On Windows these functions are in winsock.h or winsock2.h, so
> >   this was easily solved by adding an #include to tables.c
> >   (an include in flexdef.h was not possible due to a clash
> >   between the CHAR type from the windows headers and the CHAR
> >   token used in parser.c), and an #undef for Char, as Char is
> >   also a typedef used by the windows headers
> 
> who's undef'ing char?

I'm undef'ing Char (capital C).
I had originally added "#include <winsock2.h>" to flexdef.h but this
caused tons of parse errors in the standard windows headers, due to
the "#define Char unsigned char" (why not a typedef by the way?).
So i moved the include up to avoid that problem, but then ran into
parser.c which defined CHAR, causing more problems (windows headers
use both CHAR and Char as typedefs).  Since I only needed winsock2.h
for htonl and htons, I moved its include into tables.c - and since I
included it after flexdef.h, I needed to avoid the problems with the
Char define, so I "#undef Char" before including winsock2.h. tables.c
never referred to Char, so this undef is harmless.
Perhaps using a flex_char_t typedef instead of a Char define would
be cleaner (and in keeping with the integer types used).

> > - scan.c uses unistd.h unconditionally (even though configure
> >   is set up to test for its existence).  This is VERY
> >   unfortunate, as it introduces a kind of chicken-and-egg
> >   problem as a win32 flex is required to regenerate it.
> >   -> However, unistd.h seems completely unneeded in this case,
> >   so adding an '#ifndef WIN32' around it was sufficient.
> 
> scan.c cannot use configure tests. A work-around is to touch a file
> "unistd.h" in a directory where your C preprocessor will find it. But
> unistd.h is a standard POSIX header, as I recall.

It's POSIX, but not ANSI C, yes.  I will add an empty unistd.h instead
of conditionalizing the include - I agree it's a bit cleaner that way.

> > - filter.c unconditionally uses dup(), dup2() and execvp()
> >   -> On Windows these are in io.h and process.h, so easily
> >   fixed with two #includes in flexdef.h
> 
> Sounds like windows has broken headers. IT'll be easy enough to add
> configure tests to check for those headers though.

Those headers are windows-specific (windows isn't POSIX), so they
really belong in a '#ifdef WIN32' block.

> > - filter.c assigns to stdin and stdout
> >   -> this is not possible on windows (stdin is a #define for
> >   &_iob[0], so not an l-value) - since the C++ standard (and I
> >   assume the C standard too, but I don't have that handy)
> >   explicitly states stdin/stdout/stderr are macros, so that
> >   many systems happen to support this seems more an accident.
> >   Even POSIX only requires them to be macros that are
> >   pointer-to-FILE, not that they refer to assignable pointers.
> >   Perhaps this code could/should be based around libiberty's
> >   pexecute()?
> 
> Yeah, this one is a known issue and will be addressed in the near
> future. Thanks for your suggestion for an approach.
> 
> > - filter.c uses pipe()
> >   -> Windows has _pipe(), but that has a different calling
> >   interface.  This would also go away if pexecute was used as
> >   base (as this has implementations for various systems including
> >   msdos, windows, and os/2)
> 
> A possibility, but again, it looks like your real problem is that
> you're not working in a POSIX environment.

True - but DJGPP is as POSIX as it can be on a signle-tasking OS, and
therefore also does not have pipe (or fork or wait) - breaking backward
compatibility for those platforms does not seem like a good idea.

> > This only leaves the main issue, the use of fork()/wait() in
> > filter.c (and wait() in main.c, but I assume the two are related).
> > At first glance it seems as it's all for that pipe, so I'll try to
> > make an alternate implementation using popen() (which IS supported
> > on DJGPP and Windows).
> 
> If you can do things w/o fork() and wait() and friends, then go for it
> and let us know.

I'll look into it at least, as much for the sake of DJGPP as my efforts
for VS.NET - at least windows has Cygwin to fall back on.

> > If a patch for MSVC.NET support is welcome, please tell me
> > what subdir to put the windows-specific stuff in and I'll
> > send a patch once I get flex fully compiled & working.
> 
> My only issue with including separate files for any port is that I
> can't maintain them. The more that your work focuses on making flex
> more portable, the better.

This is the case for most Visual Studio support in GNU tools.
As for maintaining it, that should be trivial work for me if I can
get a working build together (barring major changes in how flex is
built of course), and I'd be quite happy to maintain them
for each flex release.

> > Oh, and since I initially downloaded the sources to see if I
> > could embed a full flex lexer (even a C style one, as I think that
> > would be easier to hook up to a bison parser) inside a
> > C++ namespace, is that possible yet?  Perhaps through a (set of)
> > "%namespace foo" or "%option namespace foo" directives?
> 
> c++ support is in a strange place now. After the upcoming 2.6 release,
> we plan to overhaul c++ support as what's present now is really just
> hacked up c++ support from about 5-7 years ago and c++ is a different
> creature now that the new standard is finally widely adopted.

At least with the new M4-based skeleton syntax it should be easier for
me to move some stuff around to get the namespace blocks where I want
them.  I'll look into it.

Thanks for the feedback.