Re: m4-1.4.x fails to build git Autoconf for some x

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

According to Keith Marshall on 3/24/2008 3:34 PM:
|> It's not clear to me from this description whether it is vanilla m4
|> that is broken, or the Autoconf freeze code (and it's just the
|> workaround that is in m4).  IOW: what would be The Right Thing to do?
|
| That's a tricky one!  Maybe a truly native m4 for Win32 really does need
| to be O_TEXT; after all that is the native standard for the platform.

One half is easy.  The right thing to do is for m4 to ALWAYS generate
O_BINARY frozen files (fixed 2006-07-06 for 1.4.5, using
fopen(name,"wb")), so that the frozen file is never generated with CR
unless the CR occurred literally in the input.  At which point, if a
frozen file is ever encountered with a CR, it must have been part of the
macro definition (which, if the input was O_TEXT, seems relatively
difficult to do, since the input never sees CR), or else inserted by the
user after the fact (perhaps by an ftp in ascii rather than binary mode).
~ In other words, I do not see how Chuck's patch to add
SKIP_CARRIAGE_RETURNS can make any difference (although it used to matter,
in 1.4.4 and earlier).

The other half also seems easy on the surface.  POSIX requires m4 to
operate on text files (as opposed to binary files); and although POSIX
also requires text and binary to be the same representation, my
interpretation is that on non-POSIX platforms where CRLF is the norm for a
text file, the file should be opened O_TEXT rather than O_BINARY.
Therefore, frozen files are currently opened the same way as all other
files, with fopen(name,"r"), rather than "rb".  This means that re-reading
the frozen file won't see CR, but if there were no CR present to begin
with, this shouldn't matter either.

|
| However, MSYS aims to provide a bridge between POSIXy build systems and
| the native Win32 environment; MSYS is strictly O_BINARY, and problems
| do seem to arise, when O_TEXT tools are mixed into the chain.

So if I understand correctly, on MSYS, fopen(name,"r") always gets
O_BINARY and sees any CR (different than cygwin, where it respects the
underlying mount point for deciding text or binary)?  Does
fopen(name,"rt") force O_TEXT as on cygwin?  However, I do not want to use
fopen(name,"rt"), as it is non-standard.

| Usually, when I'm faced with such issues, I force the input stream to be
| O_BINARY, then explicitly handle the line terminations in my own code,

Which is what the patch that Ralf pointed to did.  But I find that gross,
and prefer to use it only as a last resort, because it implies the
platform's text mode is broken.

That said, I have been debating about adding an m4 patch that opens frozen
files with O_BINARY rather than the current O_TEXT, and turns CRLF into
just LF if the version directive in the frozen file had a CRLF, on the
assumption that the user (possibly inadvertently) introduced the CR with
something like an ftp in the wrong mode.  I would use this on all
platforms, rather than just MSYS.

And there is still the question of whether M4 should continue its current
policy of O_TEXT for both normal input and output (since POSIX says it
operates on text files), or whether users want (at least an option) to
force binary processing in one or both directions.

But I wonder if the problem is not in freezing files, but in something
else.  Ralf stated:

| FWIW, with the freeze.c change alone, Autoconf still failed to build
| however.  The failure is due to "freezing produced output", notably
| lines consisting of little more than CR

Recall that running m4 -F produces output as well as a frozen file.  This
is an intentional design decision of autom4te - input is allowed to
generate blank lines to make macro manipulation simpler (think m4_define
lines that did not end in dnl), then all blank lines output to stdout are
stripped.  For most platforms, this is okay.  But on MSYS, you now have m4
generating CRLF output (since m4 does nothing special to stdout, it is
O_TEXT), but perl expecting just LF output (since MSYS perl is O_BINARY),
so what _should_ be considered a blank line isn't, and you get the warning
that there was unexpected output when trying to freeze the file.  In other
words, it sounds like autom4te should be fixed to ignore CR when ignoring
blank lines on platforms where m4 generates O_TEXT output by default.

| just killing \r in autom4te.in leads to
|
| C:\msys\1.0\home\ralf\local\bin\m4.exe: premature end of frozen file

Where did you kill \r?  From the frozen file itself (if so, how did it get
there in the first place, if m4 is reading input in O_TEXT)?

One more thought on the matter - I have been patching the master branch of
m4 to produce files with macro definitions using escape sequences rather
than literal CR or LF (although the patch is incomplete; the 'D' directive
for diversions does not do escaping yet).  This is a more robust format,
since you can then distinguish between CR that were literally part of the
macro definition and CR that were inadvertently added to the frozen file.
~ But I'm not sure if it is helpful enough to backport something like that
to m4 1.6 (and certainly not to 1.4.11).

- --
Don't work too hard, make some time for fun as well!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkfo9lsACgkQ84KuGfSFAYANxQCgrQsFglcbQLN9jak8sv5B5jWA
JPYAn25kUZK0TGY65SWpDauul54WQtAU
=CHXU
-----END PGP SIGNATURE-----