Re: [Flex] Getting the absolute input file position

[Hans Aberg]

On 16 Jul 2005, at 04:37, address@hidden wrote:

> But how come there is no yyabspos variable or feature?
>
> Shouldn't flex support that kind of 'feature' out of the box,
> why would one have to figure out that much?

It has been discussed on this list, with cons and pros. One problem  
is how to sync it with locations in the Bison generated parsers.  
Perhaps that put it off.


> > > Such that I can fopen, fseek, fread a "context" later on,
> > > after the entire parsing and processing is performed.
> >
> > Because of buffering, you can't use such functions directly.
>
> I want to use such function AFTER the parsing... not during the  
> parsing.

I don't think that will help you, because I think the Flex lexer will  
just read a chunk into a buffer, and that is where the file position  
will be after the parsing, not at actual lexing position, which then  
will be in the buffer somewhere.

> I did a search for %option:

> %option yylineno

In a lexer .c file with this option on, there is a segment right  
before the lexer switch statement that looks something like:

        YY_DO_BEFORE_ACTION;

        if ( yy_act != YY_END_OF_BUFFER && yy_rule_can_match_eol 
[yy_act] )
            {
            int yyl;
            for ( yyl = 0; yyl < yyleng; ++yyl )
                if ( yytext[yyl] == '\n' )

    yylineno++;
;
            }

Here, you should introduce a new variable besides yylineno that  
counts bytes instead, and figure out how to update it. Then put that  
alteration into your skeleton file, and make sure it is used when  
compiling the .l file with Flex.

> > So there seems to be essentially two methods,
> > first, add a byte count in each rule,
>
> As you can see, it's far from an *easy* grammar,
> so modifying each rule internal without breaking anything
> would be quite challenging.
>
> It has thousands of inter-related rules,
> and a minimally coupled 50,000 LOC,
> that's if the other flex modules are ignored from the equation
> and the calls to the "core context manipulation" is ignored also.
>
> Unless you have a way to know how much can get eated, unputed,  
> manipulated
> without adding a counter everywhere in 50KLOC.

I think that some of the proponents of not adding the feature to  
Flex, argued this was fairly straightforward; perhaps you can get  
some help from those, if the other method above does not work for  
you. :-)

> > and second, check out how %option yylineno is implemented, and
> > tweak the skeleton file for your purposes.
>
> I don't have the skeleton file  :-|

If you have Flex, you have the skeleton file, as it is used to  
produce the lexer output in ever Flex compile. In later flex  
versions, it is called flex.skl.

> However, every cPP file says:
>
> /root/flex/flex/skel.c
>
> Looks like a "default skeleton".

The problem is that you have an old Flex version, most people on this  
list have forgotten about. The latest version can be gotten from CVS  
flex.sourceforge.net, or lex.sourceforge.net (two different  
addresses, with different contents).

  Hans Aberg