Re: Fw: Does flex support UTF-8

[Jacques Beaurain]

Hi to list and Chuck Carmack,

I know this is an old topic, but just posting this here because I had
a hard time figuring out how to use Martin's patterns for exactly the
purpose of allowing UTF-8 in identifiers. Somebody else may benefit
from this later on.

Here are a few patterns that I used  to achieve it successfully.

utf8b             [\200-\277]

utf8b1            [\300-\337]{utf8b}
utf8b2            [\340-\357]{utf8b}{2}
utf8b3            [\360-\367]{utf8b}{3}
utf8b4            [\370-\373]{utf8b}{4}
utf8b5            [\374-\375]{utf8b}{5}

utf8char          ({utf8b1}|{utf8b2}|{utf8b3}|{utf8b4}|{utf8b5})

letter            [[:alpha:]]
letter_numb       [[:alnum:]_]
letter_numb_brack [[:alnum:]_\[\]]

identifier      (({letter}|{utf8char}|"_")(({letter_numb_brack}|{utf8char})*))


Cheers,
Jacques