[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: flex -- input buffer overflow problem
From: |
W. L. Estes |
Subject: |
Re: flex -- input buffer overflow problem |
Date: |
Mon, 22 Oct 2001 08:34:08 -0400 |
Anyone have any thoughts on this?
> Date: Mon, 22 Oct 2001 10:31:31 +0200
> From: Nils Nordman <address@hidden>
> User-Agent: Mutt/1.3.21i
>
> Hi,
>
> I have encountered a problem with flex that I hoped you perhaps could
> shed some light over.
>
> First off, I'm using flex as found in the current development version
> of OpenBSD, flex version 2.5.4. I've encountered the problem while
> developing a simple c-code to html converter. What that does is
> basically to recurse through a directory, scanning all files and
> match keywords, function declarations/definitions, constants,
> etc etc.
>
> Sequentially scanning lots of files produces the input buffer overflow
> error:
>
> input buffer overflow, can't enlarge buffer because scanner uses REJECT
>
> Now, I do use some really long trailing contexts. That, however, is not
> the problem. This problem will only occur if you've scanned lots of
> files previously and will not occur otherwise. In this case I use the
> scanner for the OpenBSD kernel source which is composed of 8654 files,
> although only the c and header files are scanned. With approximately 30
> files left to be scanned the scanner will bail out with the above
> error message. Just scanning the subdirectory that contains the
> offending file will not cause any problems however and replacing the
> files in that directory will still produce the same result, proving
> without doubt that the error is not a result of the trailing context.
>
> What I've done so far is that I've tried to somehow reinitialize the
> buffer before scanning each file, trying yy_flush_buffer, creating
> and using a new buffer for each file to be scanned, etc etc. All to
> no avail.
>
> I also did a debug run(with the -d flag during scanner generation) and
> noted that the scanner does behave rather strange after scanning many
> files. It accepts the default rule for things that never should be
> matched by that(one character at the time), and prints out
>
> --(end of buffer or a NUL)
>
> every other line. Then at the file for which it bails out, the only info
> printed after starting to scan it is:
>
> --(end of buffer or a NUL)
> --(end of buffer or a NUL)
> --(end of buffer or a NUL)
> input buffer overflow, can't enlarge buffer because scanner uses REJECT
>
> I've put the flex file online at http://nforced.com/users/nino/c_scanner.l
> in case you want to look at it. If you want to I can provide the debug
> output as well(although that's pretty big).
>
> I realize that this might not be enough information for giving
> any clues, but I will gladly provide any other information at your
> request.
>
> Best regards,
>
> --
> Nils Nordman <address@hidden>
>
- Re: flex -- input buffer overflow problem,
W. L. Estes <=