Re: Parsing a language with optional spaces

[Christian Schoenebeck]

On Mittwoch, 8. Juli 2020 22:14:23 CEST Akim Demaille wrote:
> > So yes, you could certainly address this to work correctly with Flex with
> > additional measures,
> 
> The case you are citing above is straightforward to handle the same
> way I did with the BASIC.  That's fairly simple given that this is
> plain rational languages, fully under the perimeter of Flex.
> 
> The general case though will not work just as well.  John reported that
> the grammar is
> 
> DO <number> <variable> = <expression> , <expression> [, <expression>]
> 
> and here the problem is that the <expression>s can have parenthesized
> expressions with embedded commas.  So on this case, Flex is screwed.
> 
> That reminds me of a paper I read long ago, someone proposing
> "heisentokens": having the scanner return multiple tokens concurrently,
> for the different scanning options, branched into GLR to follow the
> various options and let dead ends die (that would be a second time?).

"Dying branches" is exactly what I always had in mind for a potential scanner-
parser marriage in first place.

But it's probably not only about tokens in parallel; you would probably also 
need to maintain several parser states in parallel, as it might need several 
reductions to get to the point where all ambiguities are resolved and only one 
progression in the grammar remains.

But the profits would be stunning: you could write parsers in a very straight-
forward way, even for very complex languages. And on the other side it would 
eventually allow to design languages in a much more user friendly and flexible 
way as well, as it would be straight-forward e.g. to restrict keywords (or 
other patterns) to certain grammar contexts only.

Best regards,
Christian Schoenebeck