Re: common per-token info

[Hans Aberg]

At 15:36 +0100 2002/01/09, Akim Demaille wrote:
>John> Akim: Just for the sake of example, can you give us a scenario
>John> for which you need to track the "width" of each token, (where
>John> width != yyleng) ?
>
>Strings come to my mind.  You have to accumulate.  And what do you
>count: chars or bytes?

If it is variable width characters you have in your mind, discussions in
various newsgroups and lists (std.c++, LaTeX-3) suggest that one should not
use that internally in a program: Instead first translate to fixed width
characters. Then that problem goes away.

>But I might not have understood the question: if you refer to the
>width I was referring to before, I meant that there are (at least) two
>schools: one is
>
>        location = start-pos + end-pos
>
>the other being
>
>        location = start-pos + width

If, as I suggested, for location one gives [begin, end) semi-open interval
of yytext, then the width can be computed when the error occurs. So then
that issue is covered.

  Hans Aberg