[Chicken-hackers] [PATCH] Improve read/write invariance of keywords (#1332)

[Peter Bex]

Hi all,

I've taken a look at #1332.  Attached are patches against master and
CHICKEN 5 to fix keyword handling.

For instance, when reading keywords using the "portable" notation with
the #: prefix, we used r-token instead of r-xtoken.  This meant it
would not be able to deal with pipe delimiters when reading.

Keywords were written simply using symbol->string, which meant any spaces
or other special characters that needed quoting would not be treated
specially, resulting in the main issue behind #1332: keywords that cannot
be read back.

I also saw that an empty keyword could be constructed via
(string->keyword ":"), but when that was written it couldn't be read
back anymore.  Also, keywords that are written by a CHICKEN with
keyword style prefix cannot be read back by a CHICKEN with keyword
style suffix and vice versa.

To avoid all these issues, I think it's simplest to just ignore the
keyword style and _always_ write keywords using the "portable" notation.
I also removed the special case of zero-length string in "build-keyword",
because it should really support empty keywords; it's up to the reader
that precedes it to determine if it's a keyword or not.
Unfortunately, this introduces one edge case: when we've read a dot
inside a list, it may be a dotted tail notation.  When it turns out that
the value is a symbol, it will be read, but the parser doesn't know that
it has already read a character as part of this symbol.  So that's why
it needs to check if "tok" is ".:".  In case we're dealing with suffix
keyword style, this must be read as a keyword.

Our reader is rather hairy, and the patch nontrivial, but I think this
improves things enough to apply.  I'm just not 100% sure if changing the
way we write keywords should be done on master.  On one hand, it's sort
of a breaking change.  On the other hand, it isn't because old CHICKENs
can of course still read these "portable" keywords.

Cheers,
Peter

0001-Improve-read-write-invariance-of-keywords-1332.chicken-5.patch
Description: Text Data

0001-Improve-read-write-invariance-of-keywords-1332.master.patch
Description: Text Data

signature.asc
Description: Digital signature