[Bug classpath/24259] LineInputStream can cause entire underlying stream to be buffered to memory.

[daney at gcc dot gnu dot org]

------- Comment #1 from daney at gcc dot gnu dot org  2005-10-10 17:27 -------
Different (story like) discription of the problem:

LineInputStream is usually created on top of a BufferedInputStream, so the
number of bytes available will be the number bufferd in the BufferedInputStream
plus the number available in the underlying stream.

Lets say that you do a HTTP get of a large file (say 5MB) and there are 10
headers returned in the response.

Lets say that before the first call to readLine 20KB are buffered in the
kernel, LineInputStream will set the mark and read 20KB into its buffer.  But
because the BufferedInputStream needs to be able to rewind, it will buffer the
20KB also.  Now the kernel's buffers are empty, but we are reading quickly so
the kernel can get the data at the full network bandwidth (100 MB/s)  readline
returns a 40 byte header, the rest of the data is pushed back to the
BufferedInputStream.

Now it is time to read the next line.  But this time there are 20KB buffered in
the BufferedInputStream plus 200KB in the kernel.  No problem LineInputStream
allocates a 220KB buffer, sets the mark in the underlying BufferedInputStream
and reads it all in.  The BufferedInputStream now has to allocated its own
220KB buffer so that it can rewind.  The 220KB is copied into two different
buffers.  readLine returns another 40 byte header and resets the
BufferedInputStream.

Time for the next line.  This time there ate 220KB buffered in the
BufferedInputStream.  During the time we were allocating buffers and copying
data around (which was a long time since the system is heavly loaded and some
pages had to be swapped out to make room for the buffers) another 1MB of data
has been buffered in the kernel.  No problem we will just allocate two 1220KB
buffers and copy all the data around again.
.
.
.
By the time we have read all 10 headers, the entire 5MB body of the request is
buffered in memory.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=24259