Re: Memory allocation problem with latest gawk - new testcase

[Ivan Zakharyaschev]

        Hello!

On Thu, 3 Apr 2003, Aharon Robbins wrote:

> > Problem #1: gawk behaviour depends on kernel version.

> > Problem #2: memory leak (works both on 2.2.x and 2.4.x
> kernels)
> >
> > $ (ulimit -Sv 16384 && time gawk '/.*/ {;}' /dev/zero)
> > gawk: cmd. line:2: fatal: grow_iop_buffer: iop->buf: can't
> allocate 16777218 bytes of memory (Cannot allocate memory)
> > 0.03user 0.00system 0:00.12elapsed 24%CPU (0avgtext+0avgdata
> 0maxresident)k
> > 0inputs+0outputs (186major+75minor)pagefaults 0swaps
>
> And:
>
> > > Problem #2: memory leak (works both on 2.2.x and 2.4.x
> kernels)
> > >
> > > $(ulimit -Sv 16384 && time gawk '/.*/ {;}' /dev/zero)
> > > gawk: cmd. line:2: fatal: grow_iop_buffer: iop->buf: can't
> allocate 16777218 bytes of memory (Cannot allocate memory)

> > Note:
> > this testcase works as expected (endless loop) with gawk
> 3.1.1.
>
> I'm afraid you're not understanding what's going on.  The
> basic idea is that
> gawk reads characters until it finds the record separator, by
> default a newline.
> Since /dev/zero has no newlines in it, it will keep growing
> the buffer until
> it runs out of memory.  You have limited the amount of memory
> to 16 meg or so,
> so of course it eventually runs out of memory, and reports a
> fatal error.
>
> I would state that the 3.1.1 behavior is a bug, fixed in
> 3.1.2.

Actually, the first time I noticed the strange behavior of
gawk-3.1.2, there were newlines in gawk's input (coming in
reasonable periods), and the crash could be reproduced both on
2.2.24 and 2.4.20 kernels. But that case was rather complex to be
a good test case. gawk 3.1.1 didn't crash in the same conditions.
Perhaps, even if you can't reproduce it, you may have an idea
what the problem is, and how to reduce the case to a simpler one
(with enough newlines).

I ran gawk to parse emacs' output during the compilation of Emacs
Lisp libraries. To make it simpler to demonstrate the error, I
just run emacs in batch mode in a loop, and the error occurs:

 for (( i = 0; i < 10; ++i )); do emacs -q -batch -f \
list-load-path-shadows; done 2>&1 \
| awk '/!! /{ r++; } END{ if (r) printf "%d errors.\n", r; exit r; } /.*/'
...
awk: cmd. line:2: (FILENAME=- FNR=222) fatal: grow_iop_buffer:
iop->buf: can't allocate 1073741826 bytes of memory (Cannot
allocate memory)

In this case, when I first store the output of the for-loop in a
file, and then cat it into a pipe to gawk, it doesn't crash.


The output from

emacs -q -batch -f list-load-path-shadows

looks like this (I find it reasonable, if gawk doesn't need to
allocate lot of memory in order to parse it):

Loading /etc/emacs/site-start.d/00auto-compr.el (source)...
Loading /etc/emacs/site-start.d/00macros.el (source)...
Loading /etc/emacs/site-start.d/10fontlock.el (source)...
Loading /etc/emacs/site-start.d/10frames.el (source)...
Loading /etc/emacs/site-start.d/10initial-features.el (source)...
Loading /etc/emacs/site-start.d/10keys.el (source)...
Loading /etc/emacs/site-start.d/10more-cyrillic.el (source)...
Loading /etc/emacs/site-start.d/10mouse.el (source)...
Loading /etc/emacs/site-start.d/10selection.el (source)...
Loading /etc/emacs/site-start.d/20elib.el (source)...
Loading /etc/emacs/site-start.d/21eieio.el (source)...
Loading /etc/emacs/site-start.d/21speedbar.el (source)...
Loading /etc/emacs/site-start.d/22semantic.el (source)...
Loading /etc/emacs/site-start.d/23jde.el (source)...
Loading /etc/emacs/site-start.d/auctex.el (source)...
Loading /etc/emacs/site-start.d/dvilj.el (source)...
Loading /etc/emacs/site-start.d/dvips.el (source)...
Loading /etc/emacs/site-start.d/ede.el (source)...
Loading /etc/emacs/site-start.d/gettext.el (source)...
Loading /etc/emacs/site-start.d/gnats.el (source)...
Error while loading /etc/emacs/site-start.d/gnats.el
Loading /etc/emacs/site-start.d/php.el (source)...
Loading /etc/emacs/site-start.d/python.el (source)...
Loading /etc/emacs/site-start.d/ruby.el (source)...
Loading /etc/emacs/site-start.d/xdvi.el (source)...
This site has duplicate Lisp libraries with the same name.
If a locally-installed Lisp library overrides a library in the
Emacs release,
that can cause trouble, and you should probably remove the
locally-installed
version unless you know what you are doing.

/usr/share/emacs/site-lisp/auctex/style/slovak hides
/usr/share/emacs/21.3/lisp/language/slovak
/usr/share/emacs/site-lisp/auctex/style/czech hides
/usr/share/emacs/21.3/lisp/language/czech
/usr/share/emacs/site-lisp/auctex/style/swedish hides
/usr/share/emacs/21.3/lisp/international/swedish

3 Emacs Lisp load-path shadowings were found


Regards,
-- 
Ivan Zakharyaschev

JID: imz at altlinux.org