Push parsing from lexer?

[Mark Bannister]

I am writing a Bison re-entrant push parser with my own lexical analyser.  
However, if my understanding is correct, the only way I know to make this work 
is a bit inefficient.
I have a debugger interface that I'm developing that takes one line of input at 
a time.  The debugger then decides which language module to call based on a 
simple pattern match.  It invokes an entry-point in the correct module with the 
line of input it has received.  Only one of the language modules will use 
bison, the other interfaces are too simple to need it.  Once I am in the 
language module I am scanning for tokens and some simple context.  For example, 
it's possible to call out to another language module from here too so this loop 
cannot be implemented in bison as the full grammar cannot be defined at this 
point. Also, the syntax structure may be varied depending on the value of a 
variable, and this affects whether or not a complete statement is or is not 
terminated by semicolons, for example.
What I would like to do is call my bison parser for each token that I collect 
in the main loop of my language module, and from there I can create an AST.  
So, for example, I would go:
Debugger -> Module Loop -> Bison (per token) -> AST
This obviously requires a push parser, but the problem I am facing - and the 
reason for my call for help - is that bison wants to call a lexer even though I 
already have the token I need to push to it.  So it appears to be 
back-to-front.  The only way I can think of right now to solve the problem is 
to store my token in a structure that I pass by reference to bison, which it 
passes to a lexer function that just extracts the details and returns it back 
to bison.  I think this should work but it is suboptimal and looks like this:
Debugger -> Module Loop -> Bison (per token) -> lexer (return token) -> Bison 
-> AST
Is there a better way?
Best regardsMark

Sent from Yahoo Mail on Android