Memory limits?

[Thomas Worthington]

I'm testing a bit of code intended for analysing Apache Logs and hitting a 
memory limit. The log I'm using has a million lines with 18 fields each, so 
it's not completely trivial (I've already tested it on a 300 line snippet) but 
it is still an extract of the full-sized logs I want to run it on.

When the code runs I initially get a load of

"Global garbage collection... done"
"Global garbage collection... done, heap grown"
"Global garbage collection... done, heap grown"
"Global garbage collection... done, heap grown"
"Global garbage collection... done, heap grown"
"Global garbage collection... done, heap compacted"
"Global garbage collection... done, heap grown"

Which is expected, but once gst has about a gig of RAM allocated to it I get

[Memory allocation failure]
Can't allocate enough memory to continue.
done"
"Global garbage collection...

which repeats over and over (I think once per line in the log file from that 
point on). pidstat on gst at that point gives me

11:01:36      UID       PID  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
11:01:36     1000     18064      0.06      0.00  989144  813120   9.96  gst

The test box has 8G of RAM and the system I want to run it on has 680GB of RAM, 
so it would be nice to be able to use it. Is there some hard-coded limit in the 
source which I can lift or am I stuck? Or just missing something?

Thanks,
Thomas