Re: [Gluster-devel] succes sive bonnie++ tests taking lo nger and longer to run (sy stem load steadily increasing)

[Gordan Bobic]

Sounds like you are seeing the same disconnection issues I reported a while 
back.

Gordan

"Daniel Maher" <address@hidden> wrote:

>Raghavendra G wrote:
>> * What does the top output corresponding to glusterfs say? what is the 
>> memory usage and cpu usage?
>> * Do you find anything interesting in glusterfs client log files? Can we 
>> get the log files?
>
>Snapshot :
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  1185 root      20   0 2478m 909m  564 D 11.3 90.1   1064:50 glusterfs
>
>
>Over a monitored period of 30 minutes, the CPU usage hovered between 7 
>and 12, and Mem usage hovered at 90 to 91.
>
>
>
>================================================================================
>Version      : glusterfs 3.0.0 built on Feb  3 2010 14:39:23
>git: 2.0.1-886-g8379edd
>Starting Time: 2010-02-17 13:58:41
>Command line : /usr/sbin/glusterfs --log-level=NORMAL 
>--volfile=/etc/glusterfs/replicated-tcp.vol /opt/gluster
>PID          : 1185
>System name  : Linux
>Nodename     : 169.install.pxe
>Kernel Release : 2.6.26.8-57.fc8
>Hardware Identifier: i686
>
>Given volfile:
>+------------------------------------------------------------------------------+
>   1: ## file auto generated by /usr/bin/glusterfs-volgen (mount.vol)
>   2: # Cmd line:
>   3: # $ /usr/bin/glusterfs-volgen --name replicated --raid 1 
>s01:/opt/gluster s02:/opt/gluster
>   4:
>   5: # RAID 1
>   6: # TRANSPORT-TYPE tcp
>   7: volume s01-1
>   8:     type protocol/client
>   9:     option transport-type tcp
>  10:     option remote-host s01
>  11:     option transport.socket.nodelay on
>  12:     option transport.remote-port 6996
>  13:     option remote-subvolume brick1
>  14: end-volume
>  15:
>  16: volume s02-1
>  17:     type protocol/client
>  18:     option transport-type tcp
>  19:     option remote-host s02
>  20:     option transport.socket.nodelay on
>  21:     option transport.remote-port 6996
>  22:     option remote-subvolume brick1
>  23: end-volume
>  24:
>  25: volume mirror-0
>  26:     type cluster/replicate
>  27:     subvolumes s01-1 s02-1
>  28: end-volume
>  29:
>  30: volume writebehind
>  31:     type performance/write-behind
>  32:     option cache-size 4MB
>  33:     subvolumes mirror-0
>  34: end-volume
>  35:
>  36: volume readahead
>  37:     type performance/read-ahead
>  38:     option page-count 4
>  39:     subvolumes writebehind
>  40: end-volume
>  41:
>  42: volume iocache
>  43:     type performance/io-cache
>  44:     option cache-size 1GB
>  45:     option cache-timeout 1
>  46:     subvolumes readahead
>  47: end-volume
>  48:
>  49: volume quickread
>  50:     type performance/quick-read
>  51:     option cache-timeout 1
>  52:     option max-file-size 64kB
>  53:     subvolumes iocache
>  54: end-volume
>  55:
>  56: volume statprefetch
>  57:     type performance/stat-prefetch
>  58:     subvolumes quickread
>  59: end-volume
>  60:
>
>+------------------------------------------------------------------------------+
>[2010-02-17 13:58:41] W [xlator.c:655:validate_xlator_volume_options] 
>s02-1: option 'transport.remote-port' is deprecated, preferred is 
>'remote-port', continuing with correction
>[2010-02-17 13:58:41] W [xlator.c:655:validate_xlator_volume_options] 
>s01-1: option 'transport.remote-port' is deprecated, preferred is 
>'remote-port', continuing with correction
>[2010-02-17 13:58:52] E [socket.c:760:socket_connect_finish] s01-1: 
>connection to  failed (Connection refused)
>[2010-02-17 13:58:52] E [socket.c:760:socket_connect_finish] s01-1: 
>connection to  failed (Connection refused)
>[2010-02-17 13:58:58] E [socket.c:760:socket_connect_finish] s02-1: 
>connection to  failed (No route to host)
>[2010-02-17 13:58:58] E [socket.c:760:socket_connect_finish] s02-1: 
>connection to  failed (No route to host)
>[2010-02-17 14:06:23] N [client-protocol.c:6224:client_setvolume_cbk] 
>s01-1: Connected to 10.0.0.38:6996, attached to remote volume 'brick1'.
>[2010-02-17 14:06:23] N [afr.c:2625:notify] mirror-0: Subvolume 's01-1' 
>came back up; going online.
>[2010-02-17 14:06:23] N [client-protocol.c:6224:client_setvolume_cbk] 
>s01-1: Connected to 10.0.0.38:6996, attached to remote volume 'brick1'.
>[2010-02-17 14:06:23] N [afr.c:2625:notify] mirror-0: Subvolume 's01-1' 
>came back up; going online.
>[2010-02-17 14:06:33] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-17 14:06:33] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-20 20:02:43] E 
>[client-protocol.c:415:client_ping_timer_expired] s02-1: Server 
>10.0.0.39:6996 has not responded in the last 42 seconds, disconnecting.
>[2010-02-20 20:02:47] E [saved-frames.c:165:saved_frames_unwind] s02-1: 
>forced unwinding frame type(2) op(PING)
>[2010-02-20 20:02:48] N [client-protocol.c:6972:notify] s02-1: disconnected
>[2010-02-20 20:04:01] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-20 20:04:01] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-22 02:37:28] E 
>[client-protocol.c:415:client_ping_timer_expired] s02-1: Server 
>10.0.0.39:6996 has not responded in the last 42 seconds, disconnecting.
>[2010-02-22 02:37:56] E [saved-frames.c:165:saved_frames_unwind] s02-1: 
>forced unwinding frame type(1) op(READ)
>[2010-02-22 02:37:56] E [saved-frames.c:165:saved_frames_unwind] s02-1: 
>forced unwinding frame type(2) op(PING)
>[2010-02-22 02:37:56] N [client-protocol.c:6972:notify] s02-1: disconnected
>[2010-02-22 02:38:02] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-22 02:38:02] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-22 04:19:02] E 
>[client-protocol.c:415:client_ping_timer_expired] s02-1: Server 
>10.0.0.39:6996 has not responded in the last 42 seconds, disconnecting.
>[2010-02-22 04:19:50] E [saved-frames.c:165:saved_frames_unwind] s02-1: 
>forced unwinding frame type(1) op(READ)
>[2010-02-22 04:19:50] E [saved-frames.c:165:saved_frames_unwind] s02-1: 
>forced unwinding frame type(2) op(PING)
>[2010-02-22 04:19:51] N [client-protocol.c:6972:notify] s02-1: disconnected
>[2010-02-22 04:20:33] E 
>[client-protocol.c:415:client_ping_timer_expired] s01-1: Server 
>10.0.0.38:6996 has not responded in the last 42 seconds, disconnecting.
>[2010-02-22 04:20:34] E [saved-frames.c:165:saved_frames_unwind] s01-1: 
>forced unwinding frame type(1) op(READ)
>[2010-02-22 04:20:41] E [saved-frames.c:165:saved_frames_unwind] s01-1: 
>forced unwinding frame type(2) op(PING)
>[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-22 04:20:41] N [client-protocol.c:6972:notify] s01-1: disconnected
>[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] 
>s02-1: Connected to 10.0.0.39:6996, attached to remote volume 'brick1'.
>[2010-02-22 04:20:41] N [afr.c:2625:notify] mirror-0: Subvolume 's02-1' 
>came back up; going online.
>[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] 
>s01-1: Connected to 10.0.0.38:6996, attached to remote volume 'brick1'.
>[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] 
>s01-1: Connected to 10.0.0.38:6996, attached to remote volume 'brick1'.
>
>
>That is the entire client log file.  If it would be useful, i can stop 
>the tests, put the Gluster logging into the debug level, and start 
>again.  Just let me know.
>
>
>-- 
>Daniel Maher <dma+gluster AT witbe DOT net>
>
>
>_______________________________________________
>Gluster-devel mailing list
>address@hidden
>http://lists.nongnu.org/mailman/listinfo/gluster-devel

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.