Re: [Gluster-devel] io-cache exceeding cache-size value

2009/2/24 Amar (ಅಮರ್ ತುಂಬಳ್ಳಿ) <address@hidden>

Thanks for this data, Dan and Gorman.

This should help us a lot.

Regards,
Amar

2009/2/24 Gordan Bobic <address@hidden>

Just after mounting:

top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4605 root 20 0 51348 2368 1296 S 0 0.1 0:00.08 glusterfs

# kill -s SIGUSR1 4605
Arena 0:
system bytes = 479232
in use bytes = 258832
Total (incl. mmap):
system bytes = 1466368
in use bytes = 1245968
max mmap regions = 5
max mmap bytes = 1122304

After compiling the kernel:
# make clean; make dep; make -j32 all

top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4605 root 20 0 202m 97m 1316 S 0 2.5 2:42.09 glusterfs

# kill -s SIGUSR1 4605
Arena 0:
system bytes = 26423296
in use bytes = 25972704
Arena 1:
system bytes = 74113024
in use bytes = 73516560
Total (incl. mmap):
system bytes = 101523456
in use bytes = 100476400
max mmap regions = 5
max mmap bytes = 1122304

# ./pmap 4605
glusterfs(4605)
00400000 (24 KB) r-xp (09:02 442904105) /usr/sbin/glusterfsd
00606000 (4 KB) rw-p (09:02 442904105) /usr/sbin/glusterfsd
00607000 (25804 KB) rw-p (00:00 0) [heap]
40000000 (4 KB) ---p (00:00 0)
40001000 (8192 KB) rw-p (00:00 0)
40801000 (4 KB) ---p (00:00 0)
40802000 (8192 KB) rw-p (00:00 0)
3a38600000 (176 KB) r-xp (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3862c000 (2048 KB) ---p (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3882c000 (4 KB) rw-p (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3882d000 (72 KB) rw-p (00:00 0)
3d4c800000 (104 KB) r-xp (09:02 1194229765) /lib64/ld-2.5.so
3d4ca1a000 (4 KB) r--p (09:02 1194229765) /lib64/ld-2.5.so
3d4ca1b000 (4 KB) rw-p (09:02 1194229765) /lib64/ld-2.5.so
3d4cc00000 (1320 KB) r-xp (09:02 1194229772) /lib64/libc-2.5.so
3d4cd4a000 (2048 KB) ---p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4a000 (16 KB) r--p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4e000 (4 KB) rw-p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4f000 (20 KB) rw-p (00:00 0)
3d4d000000 (8 KB) r-xp (09:02 1194229776) /lib64/libdl-2.5.so
3d4d002000 (2048 KB) ---p (09:02 1194229776) /lib64/libdl-2.5.so
3d4d202000 (4 KB) r--p (09:02 1194229776) /lib64/libdl-2.5.so
3d4d203000 (4 KB) rw-p (09:02 1194229776) /lib64/libdl-2.5.so
3d4dc00000 (84 KB) r-xp (09:02 1194229786) /lib64/libpthread-2.5.so
3d4dc15000 (2044 KB) ---p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de14000 (4 KB) r--p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de15000 (4 KB) rw-p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de16000 (16 KB) rw-p (00:00 0)
3d4e400000 (28 KB) r-xp (09:02 1194229788) /lib64/librt-2.5.so
3d4e407000 (2048 KB) ---p (09:02 1194229788) /lib64/librt-2.5.so
3d4e607000 (4 KB) r--p (09:02 1194229788) /lib64/librt-2.5.so
3d4e608000 (4 KB) rw-p (09:02 1194229788) /lib64/librt-2.5.so
2aaaaaaab000 (28 KB) r-xp (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaaab2000 (2044 KB) ---p (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaacb1000 (8 KB) rw-p (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaacb3000 (576 KB) rw-p (00:00 0)
2aaaaad43000 (8 KB) r-xp (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaad45000 (2044 KB) ---p (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaaf44000 (4 KB) rw-p (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaaf4e000 (40 KB) r-xp (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaaaf58000 (2044 KB) ---p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaab157000 (4 KB) r--p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaab158000 (4 KB) rw-p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaac000000 (72380 KB) rw-p (00:00 0)
2aaab06af000 (58692 KB) ---p (00:00 0)
2b4ac185b000 (8 KB) rw-p (00:00 0)
2b4ac1866000 (404 KB) rw-p (00:00 0)
2b4ac18cb000 (104 KB) r-xp (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac18e5000 (2048 KB) ---p (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac1ae5000 (8 KB) rw-p (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac1ae7000 (64 KB) r-xp (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1af7000 (2044 KB) ---p (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1cf6000 (8 KB) rw-p (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1cf8000 (28 KB) r-xp (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1cff000 (2048 KB) ---p (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1eff000 (4 KB) rw-p (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1f00000 (108 KB) r-xp (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac1f1b000 (2044 KB) ---p (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac211a000 (8 KB) rw-p (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac211c000 (176 KB) r-xp (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac2148000 (2048 KB) ---p (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac2348000 (12 KB) rw-p (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac234b000 (56 KB) r-xp (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac2359000 (2044 KB) ---p (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac2558000 (8 KB) rw-p (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac255a000 (116 KB) r-xp (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
2b4ac2577000 (2048 KB) ---p (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
2b4ac2777000 (8 KB) rw-p (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
7fffe923a000 (84 KB) rw-p (00:00 0) [stack]
7fffe93fe000 (8 KB) r-xp (00:00 0) [vdso]
ffffffffff600000 (4 KB) r-xp (00:00 0) [vsyscall]
mapped: 207760 KB writable/private: 115848 KB shared: 0 KB

And again:
# make clean; make dep; make -j32 all

top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4605 root 20 0 266m 192m 1316 S 0 4.9 5:26.57 glusterfs

# kill -s SIGUSR1 4605

Arena 0:
system bytes = 26423296
in use bytes = 25972784
Arena 1:
system bytes = 173174784
in use bytes = 172023232
Total (incl. mmap):
system bytes = 200585216
in use bytes = 198983152
max mmap regions = 5
max mmap bytes = 1122304

# ./pmap 4605
glusterfs(4605)
00400000 (24 KB) r-xp (09:02 442904105) /usr/sbin/glusterfsd
00606000 (4 KB) rw-p (09:02 442904105) /usr/sbin/glusterfsd
00607000 (25804 KB) rw-p (00:00 0) [heap]
40000000 (4 KB) ---p (00:00 0)
40001000 (8192 KB) rw-p (00:00 0)
40801000 (4 KB) ---p (00:00 0)
40802000 (8192 KB) rw-p (00:00 0)
3a38600000 (176 KB) r-xp (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3862c000 (2048 KB) ---p (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3882c000 (4 KB) rw-p (09:02 442904212) /usr/lib64/libglusterfs.so.0.0.0
3a3882d000 (72 KB) rw-p (00:00 0)
3d4c800000 (104 KB) r-xp (09:02 1194229765) /lib64/ld-2.5.so
3d4ca1a000 (4 KB) r--p (09:02 1194229765) /lib64/ld-2.5.so
3d4ca1b000 (4 KB) rw-p (09:02 1194229765) /lib64/ld-2.5.so
3d4cc00000 (1320 KB) r-xp (09:02 1194229772) /lib64/libc-2.5.so
3d4cd4a000 (2048 KB) ---p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4a000 (16 KB) r--p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4e000 (4 KB) rw-p (09:02 1194229772) /lib64/libc-2.5.so
3d4cf4f000 (20 KB) rw-p (00:00 0)
3d4d000000 (8 KB) r-xp (09:02 1194229776) /lib64/libdl-2.5.so
3d4d002000 (2048 KB) ---p (09:02 1194229776) /lib64/libdl-2.5.so
3d4d202000 (4 KB) r--p (09:02 1194229776) /lib64/libdl-2.5.so
3d4d203000 (4 KB) rw-p (09:02 1194229776) /lib64/libdl-2.5.so
3d4dc00000 (84 KB) r-xp (09:02 1194229786) /lib64/libpthread-2.5.so
3d4dc15000 (2044 KB) ---p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de14000 (4 KB) r--p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de15000 (4 KB) rw-p (09:02 1194229786) /lib64/libpthread-2.5.so
3d4de16000 (16 KB) rw-p (00:00 0)
3d4e400000 (28 KB) r-xp (09:02 1194229788) /lib64/librt-2.5.so
3d4e407000 (2048 KB) ---p (09:02 1194229788) /lib64/librt-2.5.so
3d4e607000 (4 KB) r--p (09:02 1194229788) /lib64/librt-2.5.so
3d4e608000 (4 KB) rw-p (09:02 1194229788) /lib64/librt-2.5.so
2aaaaaaab000 (28 KB) r-xp (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaaab2000 (2044 KB) ---p (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaacb1000 (8 KB) rw-p (09:02 443222345) /usr/lib64/glusterfs/2.0.0rc1/transport/socket.so.0.0.0
2aaaaacb3000 (576 KB) rw-p (00:00 0)
2aaaaad43000 (8 KB) r-xp (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaad45000 (2044 KB) ---p (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaaf44000 (4 KB) rw-p (09:02 443222303) /usr/lib64/glusterfs/2.0.0rc1/auth/addr.so.0.0.0
2aaaaaf4e000 (40 KB) r-xp (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaaaf58000 (2044 KB) ---p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaab157000 (4 KB) r--p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaab158000 (4 KB) rw-p (09:02 1194229787) /lib64/libnss_files-2.5.so
2aaaac000000 (169120 KB) rw-p (00:00 0)
2aaab6528000 (27488 KB) ---p (00:00 0)
2b4ac185b000 (8 KB) rw-p (00:00 0)
2b4ac1866000 (404 KB) rw-p (00:00 0)
2b4ac18cb000 (104 KB) r-xp (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac18e5000 (2048 KB) ---p (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac1ae5000 (8 KB) rw-p (09:02 443252818) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/client.so.0.0.0
2b4ac1ae7000 (64 KB) r-xp (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1af7000 (2044 KB) ---p (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1cf6000 (8 KB) rw-p (09:02 443252834) /usr/lib64/glusterfs/2.0.0rc1/xlator/storage/posix.so.0.0.0
2b4ac1cf8000 (28 KB) r-xp (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1cff000 (2048 KB) ---p (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1eff000 (4 KB) rw-p (09:02 443252764) /usr/lib64/glusterfs/2.0.0rc1/xlator/features/locks.so.0.0.0
2b4ac1f00000 (108 KB) r-xp (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac1f1b000 (2044 KB) ---p (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac211a000 (8 KB) rw-p (09:02 443252823) /usr/lib64/glusterfs/2.0.0rc1/xlator/protocol/server.so.0.0.0
2b4ac211c000 (176 KB) r-xp (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac2148000 (2048 KB) ---p (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac2348000 (12 KB) rw-p (09:02 443222352) /usr/lib64/glusterfs/2.0.0rc1/xlator/cluster/afr.so.0.0.0
2b4ac234b000 (56 KB) r-xp (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac2359000 (2044 KB) ---p (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac2558000 (8 KB) rw-p (09:02 443252786) /usr/lib64/glusterfs/2.0.0rc1/xlator/mount/fuse.so.0.0.0
2b4ac255a000 (116 KB) r-xp (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
2b4ac2577000 (2048 KB) ---p (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
2b4ac2777000 (8 KB) rw-p (09:02 442904974) /usr/lib64/libfuse.so.2.7.4
7fffe923a000 (84 KB) rw-p (00:00 0) [stack]
7fffe93fe000 (8 KB) r-xp (00:00 0) [vdso]
ffffffffff600000 (4 KB) r-xp (00:00 0) [vsyscall]
mapped: 273296 KB writable/private: 212588 KB shared: 0 KB

I hope this is useful.

Gordan

Amar Tumballi (bulde) wrote:

Hi Gordan and Dan,

It would help me a lot if its possible for you to get the info as described below,

compile glusterfs like

bash# make clean > /dev/null
bash# make CFLAGS="-g -O0 -DDEBUG" > /dev/null
bash# make install

run the process which consumes memory (mostly client process) like below:

bash# glusterfs <any argument you give generally> -N
<this process will run in foreground now>

Open another terminal

bash# ps aux | grep glusterfs
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check in other terminal for memory usage stats>

bash# <run your application over glusterfs as you do till you get high memory usage of glusterfs.. >
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check the stat in another terminal>

bash# echo 3 > /proc/sys/vm/drop_caches
bash# kill -s SIGUSR1 <pid of glusterfs -N process>
<Check the stat in another terminal>

Even after dropping caches, if 'in use bytes =' in malloc stats shows a high value, then it is a leak. If its showing less, but just 'system bytes = ' is a high value, this means glusterfs is not really consuming high memory, but the problem is really in the memory allocation segments.

Regards,
Amar

NOTE: 'malloc_stats' will be printed to 'stdout' if we enable -DDEBUG while compiling glusterfs, as it hits performance badly otherwise.

2009/2/23 Gordan Bobic <address@hidden <mailto:address@hidden>>

Dan Parsons wrote:

I'm having an issue with glusterfs exceeding its cache-size
value. Right now I have it set to 4000MB and I've seen it climb
as high as 4800MB. If I set it to 5000, I've seen it go as high
as 6000MB. This is a problem because it causes me to set the
value very low so that my apps don't get pushed into swap. Is
there any way to fix this? To get it to stick to the limit I set
and not exceed?

It's possible you are running into the same memory leak that I'm
seeing, and I'm not using io-cache or any other performance
translators at all. With rootfs on Gluster, doing a kernel compile
(kernel source tree being on NFS, so this won't be contributing to
the bloat, hopefully) makes the glusterfsd bloat by about 80MB per
pass, and never frees it.

Gordan

_______________________________________________
Gluster-devel mailing list
address@hidden <mailto:address@hidden>

http://lists.nongnu.org/mailman/listinfo/gluster-devel

--
Amar Tumballi
Gluster/GlusterFS Hacker
[bulde on #gluster/irc.gnu.org <http://irc.gnu.org>]

http://www.zresearch.com - Commoditizing Super Storage!

_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel

--
Amar Tumballi
Gluster/GlusterFS Hacker
[bulde on #gluster/irc.gnu.org]

http://www.zresearch.com - Commoditizing Super Storage!

_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel

From:	Dan Parsons
Subject:	Re: [Gluster-devel] io-cache exceeding cache-size value
Date:	Wed, 4 Mar 2009 11:22:18 -0800