Re: Which lexer do people use?

[Derek Clegg]

On Jul 3, 2020, at 11:14 PM, Akim Demaille <akim@lrde.epita.fr> wrote:
> 
> Hi Daniele,
> 
>> Le 3 juil. 2020 à 23:15, Daniele Nicolodi <daniele@grinta.net> a écrit :
>> 
>> Hello,
>> 
>> the historical pairing is using Flex with Bison. However, while Bison is
>> under active development and seems to be a very solid code base, there
>> isn't much activity on the Flex side https://github.com/westes/flex and
>> Flex codebase and capabilities show their age.
> 
> Yes.  I have a couple of issues opened over there, and it takes for ages
> to get them processed.  When they are.
> 
> When I tried to modernize the Flex doc about Bison, they even managed to
> turn this into a lecture about software maintenance.  And not install
> my changes.
> 
> https://github.com/westes/flex/pull/420

My experience as well. I have found them to be unhelpful, slow, and 
argumentative. At times, I’ve thought about branching flex and cleaning up the 
obvious problems, but — since it’s really not that hard to write a lexer — I 
just roll my own. This is much better and it integrates well with bison, unlike 
the hoops needed when I use flex.

> 
>> I recently became aware of RE/flex https://www.genivia.com/reflex.html
>> which seems very promising. However, it only generates a C++ scanner
>> which may be (I haven't tried) to retro-fit into existing C projects to,
>> for example, gain full unicode (in its utf8 encoded form) support.
> 
> It seems amazing.  Featurewise and performancewise.  I did not know it
> (nor did I know ugrep).
> 
> I've seen projects use ragel (http://www.colm.net/open-source/ragel/)
> and re2c (https://re2c.org).  But, sadly, I have first-hand experience
> with Flex only, I can't comment about the others.

This is very interesting. It’s good for me since I pretty much only use C++.

>> Has anyone tried to hammer a C++ scanner peg generated by RE/flex into a
>> C grammar hole generated by Bison?

Let us know how it works out if you try!

Derek