Re: Memory allocation problems glibc-2.2.2

[Wolfram Gloger]

Hello,

> Our image caching code is causing memory usage to explode.  The problem 
> is that a lot of allocations are being done of the pattern:
> 
> uchar* bigData = new uchar[ 128 * 128 * 3 ]; // Alloc 128x128 rgb tile
> int* cacheId = new int;       // Create cache id for lookup
> 
> Doing this in a loop causes memory usage to explode.

Please consider

 JM Robson.
 An estimate of the store size necessary for dynamic storage allocation.
 Journal of the ACM , 18(3):416-423, July 1971.

There it has been proven that for _any_ memory allocator there exist
allocation patterns that will show fragmentation (waste of memory) _at
least_ propotional to

 log_2 (size of largest object/size of smallest object)

For your example case, this _lower bound_ actually increases steadily
as your largest object size increases, while the smallest size stays
constant.

> Btw, I reported this about 8 months ago when this was observed with 
> redhat 6.2 (glibc 2.1.3) and was told that it would be fixed.  Now I'm 
> on redhat 7.1 with glibc 2.2.2and this still hasn't been fixed.

If we would have `fixed' this case, you (or someone else) would
quickly have pointed out another slightly adapted case that would have
shown similar behaviour.  There is _no_ general solution to this
problem!

However, I believe there is an easy special-case solution in the glibc
allocator at least for the code fragment above: Please try
mallopt(M_MMAP_THRESHOLD, 128*128*3) at the start of your program.
Then it will be guaranteed that all those large chunks are returned to
the system at free() time.  The (equivalent) setting of the
environment variable MALLOC_MMAP_THRESHOLD_=49152 also allowed me to
run your test program to completion.

Regards,
Wolfram.