frozen files format 2

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In the "sysval and doc fixes" thread:
> But it does beg the question - should frozen files preserve sysval?
> It's not worth changing this for 1.4.x, but we should probably add
> another command to frozen file format 2 in m4-2.0 to preserve nonzero
> sysval over freezing.

Another thing that should be preserved over freezing is m4wrap text.
Right now:

$ echo "m4wrap(\`hi
')dnl" > a.m4
$ echo "define(\`hi', \`HI')dnl" > b.m4
$ m4 a.m4 b.m4
HI
$ m4 -F a.m4f a.m4
hi
$ m4 -R a.m4f b.m4
$

Oops - the semantics of a followed by b in the same m4 run are not met,
since m4wrap was expanded during freezing instead of saved for use after
end of input on reload.  I propose that version 2 of frozen files adds the
"'W' length '\n' string '\n'" command, which adds the string to the end of
the m4wrap text (assuming we also fix m4wrap to be FIFO).

Back to the topic of sysval, just adding a command to set just sysval is
not scalable; who knows what other modules will have state to save,
although we don't want to add a command letter for every module.  Instead,
I propose extending the M command.  Right now, M only takes a single
argument, the module name to load.  I propose the alternative "'M' length1
',' length2 '\n' string1 string2 '\n'" command, generated on freezing if a
module has a freeze hook installed, and on reload, feed string2 to the
reload hook of the module (if a reload hook is present); if a module has a
reload hook but there is no string2, then call the reload hook with the
empty string.  Then, the m4 module, which provides the sysval stateful
macro, would also provide a freeze hook which outputs the value of sysval
as a string, and a reload hook which parses the string and sets the value
of sysval.

This means that m4module.h would need to be expanded to add
M4FREEZE_HANDLER (called when a module is being frozen, to give the module
a chance to output a string according to its own formatting conventions of
state to preserve), and M4RELOAD_HANDLER (called when a module is being
reloaded, to give the module to restore frozen state).

My next question concerns text builtins like __unix__.  Right now, the gnu
module determines at compile-time whether __unix__ or __windows__ should
be defined.  But a frozen file created on one type of machine, and
reloaded on another, will result in the wrong macro being defined.  Is
this okay, or should we be adding some smarts to the module freezing which
notices when __unix__ was never redefined by the user on a unix machine,
omit it from the frozen file; and on reload on a windows machine,
instantiate __windows__ if neither one of the two were explicitly
mentioned in the frozen file?

Next, should we allow partial input?  Right now, m4 chokes if a file is in
the middle of a macro, even if the next file could complete the story:
$ echo "len(" > c.m4
$ echo "abc)" > d.m4
$ m4 c.m4 d.m4
NONE:0: m4: ERROR: EOF in argument list
$ cat c.m4 d.m4 | m4
3
$ m4 -F c.m4f c.m4
NONE:0: m4: ERROR: EOF in argument list
$

But since we also refuse to create c.m4f, our behavior of freezing matches
our normal behavior.  I'll have to check how Solaris behaves in this case,
but right now, I don't think I'm inclined to change behavior here.

Then there is the question of diversion 0.  Right now, the frozen file
format supports reloading diversion 0, which dumps to stdout; but freezing
never generates a diversion 0 in the frozen file (instead, m4 already
printed the text to stdout prior to the creation of the frozen file).  So
an application that wants to preserve output is permitted to do something
like:
$ m4 -F foo.m4f foo.m4 > foo.out
$ printf 'D0,%d\n' $(wc -c < foo.out) >> foo.m4f
$ cat foo.out >> foo.m4f
$ echo >> foo.m4f

But it might be nice if we added an option to m4, perhaps named
- --freeze-stdout, which is only valid if -F is also in effect, and which
tells m4 to buffer diversion 0 into the frozen file rather than straight
to stdout.

Finally, I propose tightening the syntax of frozen files.  Right now, the
"'V' number '\n'" command can occur anywhere, multiple times or not at
all, but some of the reload logic in CVS head depends on knowing whether
we are version 1 or 2.  I propose that the frozen file format be changed
to require that 'V' always be present as the first non-comment command,
and reject a frozen file as invalid if some other command appears first or
if V appears more than once.  This should be upwards compatible with
existing frozen files, since the output generated by m4 -F already meets
this, and it is not likely that many people have hand-edited frozen files
to violate this constraint.  Of all the ideas in this email, this is the
only idea that we should also fold into m4 1.4.5.

- --
Life is short - so eat dessert first!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEmpMx84KuGfSFAYARAltKAKDHunRtKauLDIkOrYsjnm2xacec3wCfWdip
MLI/YorzJDdVnSoVPpvv4vw=
=AefF
-----END PGP SIGNATURE-----