Re: [Gluster-devel] Excessive memory usage with 1.3.12

[Dan Parsons]

To be fair, I'm not even sure it's technically a "leak", it's just io- 
cache not limiting itself to the cache-size setting. As in, io-cache  
is doing its job, it's just doing it a bit .. too enthusiastically.

First, here's my io-cache block:

volume ioc
  type performance/io-cache
  subvolumes stripe0         # In this example it is 'client' you may  
have to change it according to your spec file.
  option page-size 1MB      # 128KB is default
  option cache-size 2048MB    # 32MB is default
  option force-revalidate-timeout 5 # 1second is default
  option priority *.psiblast:3,*.seq:2,*:1
end-volume

And for my test:

address@hidden ~]# cd /distfs
address@hidden distfs]# ls -lah bigfile.img
-rw-r--r-- 1 root employees 6.3G Aug 26 00:21 bigfile.img

Note the memory usage of a freshly started glusterfs client:
root     23972  0.0  0.0 110340  1540 ?        Ssl  11:37   0:00  
[glusterfs]

Now I do this:
address@hidden distfs]# cat bigfile.img > /dev/null

The glusterfs memory footprint grows at a nice rate. In just a few  
seconds it's gone from 110kb to 2.5GB, way past my cache-size limit.

root     23972  9.4 61.8 2590552 2503164 ?     Ssl  11:37   0:14  
[glusterfs]

This box has only 4GB RAM so i killed the 'cat' process before things  
got out of hand. But, there's your test.

address@hidden ~]# glusterfs --version
glusterfs 1.3.11 built on Aug 21 2008 11:26:38
Repository revision: glusterfs--mainline--2.5--patch-795


Dan Parsons


On Nov 7, 2008, at 11:31 AM, Krishna Srinivas wrote:

> Lukas, Dan,
>
> Are you sure that its a leak of io-cache? did you try by removing the
> translator and not observe the leak?
>
> What made you conclude that cache-size option was being ignored? It
> could be a memory leak by io-cache at some other place too.
>
> I tried to make io-cache memleak, but its not happening, what
> operations do you guys do to see the leak? can you see if some simple
> test case makes it leak? (so that it will be easy for me to reproduce
> the bug here and fix it)
>
> Thanks!
> Krishna
>
> On Wed, Nov 5, 2008 at 11:38 PM, Dan Parsons <address@hidden>  
> wrote:
> > Lukas, just to confirm your findings, I have the exact same problem  
> > and
> > reported it about 2 months ago. Just like you, when all my stuff  
> > was running
> > under 32-bit, it wasn't an issue because of the 2GB limit, but now  
> > that I'm
> > using 64-bit for everything, it is a potential system crashing bug.
> >
> >
> > Dan Parsons
> >
> >
> > On Nov 5, 2008, at 1:05 AM, Lukas Hejtmanek wrote:
> >
> > > Hello,
> > >
> > > On Tue, Nov 04, 2008 at 12:37:03PM +0530, Krishna Srinivas wrote:
> > > > We want to reproduce the leak in our setup to fix it. What is your
> > > > setup on the client side? How many servers do you have? What are  
> > > > the
> > > > applications you run on the mount point? Do you observe leak only  
> > > > when
> > > > "certain" operations are done? (I am just looking for more clues)
> > >
> > > upon my investigation, it seems that the io-cache translator  
> > > ignores the
> > > cache
> > > size config option and on 64bit systems is can easily eat whole  
> > > memory. On
> > > 32bit, it usually fails to mmap additional data due to 2GB limit to
> > > process
> > > address space.
> > >
> > > --
> > > Lukáš Hejtmánek
> > >
> > >
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > address@hidden
> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel