Brent,
The 'leak' has been plugged in the read-ahead translator of the
latest tla patch. the problem was an integer overflow issue where the
'size' parameter for the flush function was 32bit but when the file
was bigger than 4GB, the code tried to flush a region from 0 to >4GB.
changing the function prototype to have a 64bit size fixed it.
The 'leak' was actually not a 'leak' because when the file was closed,
everything was flushed and free'd.
thanks!
avati
On Mon, Mar 12, 2007 at 02:15:31PM -0400, Brent A Nelson wrote:
Attached are my present spec files for both servers (which are also my
clients); both machines use the same files.
To trigger the memory leak, all I need to do is a "dd if=/phys/blah0
of=/dev/null bs=1M", where /phys/blah0 was created with "dd if=/dev/zero
of=/phys/blah0 bs=10M count=1024". It seems fine for a moment, but then
glusterfs starts growing and growing until it can't claim any more RAM
and becomes useless (can't ls the mountpoint, dd doesn't complete)...
Both machines are running Ubuntu Edgy + Fuse 2.6.3.
Thanks,
Brent
On Fri, 9 Mar 2007, Anand Avati wrote:
Brent,
we are unable to get the memory leak in read-ahead (we found one and
fixed, but your mail indicateds there is another). can you please mail
your exact volume spec files and the command line to initiate the I/O
which reproduces the leak?
regards,
avati
On Fri, Mar 09, 2007 at 07:25:03PM -0500, Brent A Nelson wrote:
I just confirmed it on client-side, as well. glusterfs was stable for a
minute, and then it suddenly started increasing in size very rapidly.
Thanks,
Brent
volume disk0
type storage/posix # POSIX FS translator
option directory /share0 # Export this directory
end-volume
volume disk1
type storage/posix # POSIX FS translator
option directory /share1 # Export this directory
end-volume
volume disk2
type storage/posix # POSIX FS translator
option directory /share2 # Export this directory
end-volume
volume disk3
type storage/posix # POSIX FS translator
option directory /share3 # Export this directory
end-volume
volume disk4
type storage/posix # POSIX FS translator
option directory /share4 # Export this directory
end-volume
volume disk5
type storage/posix # POSIX FS translator
option directory /share5 # Export this directory
end-volume
volume disk6
type storage/posix # POSIX FS translator
option directory /share6 # Export this directory
end-volume
volume disk7
type storage/posix # POSIX FS translator
option directory /share7 # Export this directory
end-volume
volume disk8
type storage/posix # POSIX FS translator
option directory /share8 # Export this directory
end-volume
volume disk9
type storage/posix # POSIX FS translator
option directory /share9 # Export this directory
end-volume
volume disk10
type storage/posix # POSIX FS translator
option directory /share10 # Export this directory
end-volume
volume disk11
type storage/posix # POSIX FS translator
option directory /share11 # Export this directory
end-volume
volume disk12
type storage/posix # POSIX FS translator
option directory /share12 # Export this directory
end-volume
volume disk13
type storage/posix # POSIX FS translator
option directory /share13 # Export this directory
end-volume
volume disk14
type storage/posix # POSIX FS translator
option directory /share14 # Export this directory
end-volume
volume disk15
type storage/posix # POSIX FS translator
option directory /share15 # Export this directory
end-volume
volume share0
type performance/io-threads
option thread-count 8
subvolumes disk0
end-volume
volume share1
type performance/io-threads
option thread-count 8
subvolumes disk1
end-volume
volume share2
type performance/io-threads
option thread-count 8
subvolumes disk2
end-volume
volume share3
type performance/io-threads
option thread-count 8
subvolumes disk3
end-volume
volume share4
type performance/io-threads
option thread-count 8
subvolumes disk4
end-volume
volume share5
type performance/io-threads
option thread-count 8
subvolumes disk5
end-volume
volume share6
type performance/io-threads
option thread-count 8
subvolumes disk6
end-volume
volume share7
type performance/io-threads
option thread-count 8
subvolumes disk7
end-volume
volume share8
type performance/io-threads
option thread-count 8
subvolumes disk8
end-volume
volume share9
type performance/io-threads
option thread-count 8
subvolumes disk9
end-volume
volume share10
type performance/io-threads
option thread-count 8
subvolumes disk10
end-volume
volume share11
type performance/io-threads
option thread-count 8
subvolumes disk11
end-volume
volume share12
type performance/io-threads
option thread-count 8
subvolumes disk12
end-volume
volume share13
type performance/io-threads
option thread-count 8
subvolumes disk13
end-volume
volume share14
type performance/io-threads
option thread-count 8
subvolumes disk14
end-volume
volume share15
type performance/io-threads
option thread-count 8
subvolumes disk15
end-volume
volume server
type protocol/server
option transport-type tcp/server # For TCP/IP transport
# option transport-type ib-sdp/server # For Infiniband transport
# option bind-address 192.168.1.10 # Default is to listen on all interfaces
# option listen-port 6996 # Default is 6996
option client-volume-filename /etc/glusterfs/glusterfs-client.vol
subvolumes share0 share1 share2 share3 share4 share5 share6 share7 share8
share9 share10 share11 share12 share13 share14 share15
# NOTE: Access to any volume through protocol/server is denied by
# default. You need to explicitly grant access through # "auth"
# option.
option auth.ip.share0.allow 128.227.64.*,128.227.89.*
option auth.ip.share1.allow 128.227.64.*,128.227.89.*
option auth.ip.share2.allow 128.227.64.*,128.227.89.*
option auth.ip.share3.allow 128.227.64.*,128.227.89.*
option auth.ip.share4.allow 128.227.64.*,128.227.89.*
option auth.ip.share5.allow 128.227.64.*,128.227.89.*
option auth.ip.share6.allow 128.227.64.*,128.227.89.*
option auth.ip.share7.allow 128.227.64.*,128.227.89.*
option auth.ip.share8.allow 128.227.64.*,128.227.89.*
option auth.ip.share9.allow 128.227.64.*,128.227.89.*
option auth.ip.share10.allow 128.227.64.*,128.227.89.*
option auth.ip.share11.allow 128.227.64.*,128.227.89.*
option auth.ip.share12.allow 128.227.64.*,128.227.89.*
option auth.ip.share13.allow 128.227.64.*,128.227.89.*
option auth.ip.share14.allow 128.227.64.*,128.227.89.*
option auth.ip.share15.allow 128.227.64.*,128.227.89.*
end-volume
volume share0-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share0 # name of the remote volume
end-volume
volume share0-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share0 # name of the remote volume
end-volume
volume share1-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share1 # name of the remote volume
end-volume
volume share1-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share1 # name of the remote volume
end-volume
volume share2-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share2 # name of the remote volume
end-volume
volume share2-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share2 # name of the remote volume
end-volume
volume share3-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share3 # name of the remote volume
end-volume
volume share3-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share3 # name of the remote volume
end-volume
volume share4-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share4 # name of the remote volume
end-volume
volume share4-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share4 # name of the remote volume
end-volume
volume share5-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share5 # name of the remote volume
end-volume
volume share5-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share5 # name of the remote volume
end-volume
volume share6-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share6 # name of the remote volume
end-volume
volume share6-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share6 # name of the remote volume
end-volume
volume share7-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share7 # name of the remote volume
end-volume
volume share7-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share7 # name of the remote volume
end-volume
volume share8-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share8 # name of the remote volume
end-volume
volume share8-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share8 # name of the remote volume
end-volume
volume share9-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share9 # name of the remote volume
end-volume
volume share9-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share9 # name of the remote volume
end-volume
volume share10-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share10 # name of the remote volume
end-volume
volume share10-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share10 # name of the remote volume
end-volume
volume share11-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share11 # name of the remote volume
end-volume
volume share11-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share11 # name of the remote volume
end-volume
volume share12-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share12 # name of the remote volume
end-volume
volume share12-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share12 # name of the remote volume
end-volume
volume share13-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share13 # name of the remote volume
end-volume
volume share13-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share13 # name of the remote volume
end-volume
volume share14-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share14 # name of the remote volume
end-volume
volume share14-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share14 # name of the remote volume
end-volume
volume share15-0
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.241 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share15 # name of the remote volume
end-volume
volume share15-1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
# option transport-type ib-sdp/client # for Infiniband transport
option remote-host 128.227.64.242 # IP address of the remote brick
# option remote-port 6996 # default server port is 6996
option remote-subvolume share15 # name of the remote volume
end-volume
volume mirror0
type cluster/afr
subvolumes share0-0 share0-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror1
type cluster/afr
subvolumes share1-1 share1-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror2
type cluster/afr
subvolumes share2-0 share2-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror3
type cluster/afr
subvolumes share3-1 share3-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror4
type cluster/afr
subvolumes share4-0 share4-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror5
type cluster/afr
subvolumes share5-1 share5-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror6
type cluster/afr
subvolumes share6-0 share6-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror7
type cluster/afr
subvolumes share7-1 share7-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror8
type cluster/afr
subvolumes share8-0 share8-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror9
type cluster/afr
subvolumes share9-1 share9-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror10
type cluster/afr
subvolumes share10-0 share10-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror11
type cluster/afr
subvolumes share11-1 share11-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror12
type cluster/afr
subvolumes share12-0 share12-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror13
type cluster/afr
subvolumes share13-1 share13-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror14
type cluster/afr
subvolumes share14-0 share14-1
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirror15
type cluster/afr
subvolumes share15-1 share15-0
option replicate *:2 # Do not leave space before or after "," and ":"
end-volume
volume mirrors
type cluster/unify
subvolumes mirror0 mirror1 mirror2 mirror3 mirror4 mirror5 mirror6 mirror7
mirror8 mirror9 mirror10 mirror11 mirror12 mirror13 mirror14 mirror15
# option scheduler rr
option scheduler alu
option alu.order
disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
option alu.disk-usage.entry-threshold 2GB
option alu.disk-usage.exit-threshold 60MB
option alu.limits.min-free-disk 5GB
option alu.stat-refresh.interval 10sec
end-volume
volume statprefetch
type performance/stat-prefetch
option cache-seconds 2
subvolumes mirrors
end-volume
volume writebehind
type performance/write-behind
option aggregate-size 131072 # in bytes
subvolumes statprefetch
end-volume
volume readahead
type performance/read-ahead
option page-size 65536 ### in bytes
option page-count 16 ### memory cache size is page-count x page-size per file
subvolumes writebehind
end-volume
--
Shaw's Principle:
Build a system that even a fool can use,
and only a fool will want to use it.