Re: [Gluster-devel] GlusterFS Process Growing

[Filipe Maia]

Could you try reproducing this inside a glusterfs chrooted dir? This
way you would be able to get glusterfs log files and still use
glusterfs for the root. Logging with debug would be good.

On Wed, Feb 11, 2009 at 21:10, Gordan Bobic <address@hidden> wrote:
> I just did some more testing, and the glusterfs (rootfs one) process grows
> like crazy when compiling the kernel - even if the said kernel isn't on the
> said rootfs, but on an external NFS share!
>
> I can think of only two things that this could be related to:
> 1) shared library loading/lookups
> 2) Lots of files being created and then deleted (compiler keeps lots of
> files in /tmp/ which is on glusterfs)
>
> Either way, it seems that deleted files seem to have handles or some such
> leaked in the glusterfs process (dcache or elsewhere). The problem is very
> real and very repeatable. I have just seen my rootfs glusterfs process grow
> from 60MB to 310MB and it's still growing. It grows at a rate of about 100MB
> per kernel compilation.
>
> I'm going to test this again with /tmp/ on a local FS to see if it that
> makes a difference.
>
> Gordan
>
> Gordan Bobic wrote:
>>
>> On Tue, 10 Feb 2009 20:25:59 +0530, Anand Avati <address@hidden>
>> wrote:
>>>>
>>>> Is there a parameter to limit this cache? And how does this compare to
>>>> the normal page cache on the underlying FS?
>>>
>>> It is not possible to limit this cache as it is decided by the dcache
>>> pruning algorithm in the linux kernel vm. You can however force a
>>> forget of all the build cache by doing an 'echo 3 >
>>> /proc/sys/vm/drop_caches'
>>
>> Interesting, I thought the drop_caches didn't include the caches that
>> reflect in the process size itself. So the 300MB of the resident size of
>> glusterfs gets included? I'll have to try this. So, running
>>
>> # updatedb
>>
>> and/or
>>
>> # ls -laR /
>>
>> should make it go up to maximum size and not grow much further?
>>
>>>> The thing that seems particularly odd is that although both the root and
>>>> the /home glusterfs daemons seem to grow over time, the growth of the
>>>> root
>>>> one seems to be vastly greater. The rootfs only contains about 1.2GB of
>>>> data, but I have seen it's daemon process get to nearly 300MB. The /home
>>>> FS
>>>> contains several hundred gigabytes of data, but I haven't seen it grow
>>
>> to
>>>>
>>>> more than about 60MB. Since the /home FS gets a lot more r/w access, it
>>>> seems to be something specific to the rootfs usage that causes the
>>>> process
>>>> bloat. Perhaps something like shared libraries (all of which are on the
>>>> rootfs)? Maybe some kind of a leak in mmap() related code?
>>>
>>> Memory usage of GlusterFS is largely decided by the dcache, which is a
>>> reflection of how many files/directories are present in your volume
>>> (and not related to the data size).
>>
>> OK, thanks for clarifying.
>>
>> Gordan
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>