Re: [Flex] Getting the absolute input file position

[f_plouff]

Hi Hans Aberg,

Thanks for the quick reply.

> > I need to figure out how I can augment the generated file to get
> > the absolute input file position within two rules:
> > start file position...end file position.

But how come there is no yyabspos variable or feature?

Shouldn't flex support that kind of 'feature' out of the box,
why would one have to figure out that much?

It's much easier if the compiler-compiler, provides as much
context information and statistics than having the resulting code
trying to figure out where we are, etc.

I mean how can I know how much get unput(), eated or skipped *easily* ?
basically know where I am from the start without touching
everything, everywhere. Unless you have an easy solution.

> > Such that I can fopen, fseek, fread a "context" later on,
> > after the entire parsing and processing is performed.
>
> Because of buffering, you can't use such functions directly.

I want to use such function AFTER the parsing... not during the parsing.
In fact I don't want to touch the parser as much as I can,
so I don't break anything that I really don't fully understand yet.
I have a good idea of how it works, but the details of why/how
and corner cases are out of reach for now.

I did a search for %option:

%option noyywrap

In code.l, commentcnv.l, commentscan.l, pre.l, scanner.l

The function getCodeFragment() which is line based,
not absolute position based is using the above,
as far as I could figure out.

and

%option nounput
%option noyywrap

in config.l, constexp.l, declinfo.l, defargs.l

and

%option noyywrap
%option yylineno

in tokenizer.l

and no %option in tag.l



These .l files are:
3070 LOC, 550 LOC, 2115 LOC, 2400 LOC and 4600 LOC respectively.

The generated .cpp files are:
11540 LOC, 2580 LOC, 6510 LOC, 5950 LOC and 14300 LOC respectively. :-S

with something between 100 and 600 rules each.



getCodeFragment() basically reopen, reread and reparse the file
using a *broken* partial parser and output start line to end line
of the input source code into a string buffer.

Not only it's time consuming for nothing [full reread/reparse],
it's also very inaccurate!!! Since the output ends up creating false positive,
because of the line versus absolute position inaccuracies for corner case.

Time is barely an issue, even though our compile bench takes over 8 hours,
mostly I/O disk seek bottleneck, since we were able to optimize it down
to 3 hours using ramdisk for I/O for input source of 200+ KLOC.

> So there seems to be essentially two methods,
> first, add a byte count in each rule,

As you can see, it's far from an *easy* grammar,
so modifying each rule internal without breaking anything
would be quite challenging.

It has thousands of inter-related rules,
and a minimally coupled 50,000 LOC,
that's if the other flex modules are ignored from the equation
and the calls to the "core context manipulation" is ignored also.

Unless you have a way to know how much can get eated, unputed, manipulated
without adding a counter everywhere in 50KLOC.

At first, I was using a mix of yy_n_chars, yytext, yyleng and similar
with a context stack, but that didn't work quite well.

The easiest way would be before and after the "switch(yy_act)".


> and second, check out how %option yylineno is implemented, and
> tweak the skeleton file for your purposes.

I don't have the skeleton file  :-|

However, every cPP file says:

/root/flex/flex/skel.c

Looks like a "default skeleton".

> > Is there a mailing list for GNU flex?
>
> I think this list is as much Flex there is, but strictly speaking, it
> isn't distributed under a GNU copyright, but a BSD one.
>
>    Hans Aberg

Sincerely yours,
Frederic Plouffe, B.Eng.