Re: User Token Numbers

[Paul Eggert]

> From: Akim Demaille <address@hidden>
> Date: 08 Apr 2002 19:20:17 +0200
> 
> While writing test cases I fell on some weird behaviors: 1.28 (I don't
> know for 1.3x, I don't have it at hand, but I'd say it behaves
> equally) accepts that you
> 
> %token FOO 2
> 
> although 2 is used for $undefined.

That is a clear violation of POSIX, I'm afraid.

> But I would like to know more from POSIX, in particular, I
> would like to make sure that EOF (= 0) is the only number that is
> reserved.

It depends on what you mean by "reserved".  If you you mean "yacc
cannot assign these token numbers", then POSIX reserves all
nonpositive integers, all token numbers used for literals in the
grammar, and all user-assigned token numbers.

Here are the relevant quotes:

 * "The end of the input is marked by a special token called the
   endmarker, which has a token number that is zero or negative. (These
   values are invalid for any other token.) All lexical analyzers shall
   return zero or negative as a token number upon reaching the end of
   their input."

 * "All token numbers assigned by yacc shall be unique and distinct
   from the token numbers used for literals and user-assigned tokens."

Other things from POSIX include:

 * User-defined assigned token numbers must be positive.

 * Yacc can assign a token number immediately when it sees a %token
   declaration.  However, in my opinion it'd be a bit nicer if it
   waits till end of grammar, so that it doesn't assign a token number
   that collides with a later user-defined token number.

 * Users can't change a token number after it has been assigned.

 * Literals must be single-byte.  "Multi-byte characters should be
   recognized by the lexical analyzer and returned as tokens. They
   should not be returned as multi-byte character literals."

 * "If duplicate token numbers cause conflicts in parser generation,
   yacc shall report an error; otherwise, it is unspecified whether the
   token assignment is accepted or an error is reported."