Re: [Gluster-devel] Mounting with file servers is failing very frequentl

gluster-devel

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] Mounting with file servers is failing very frequentl

From:	Joseph Job
Subject:	Re: [Gluster-devel] Mounting with file servers is failing very frequently (in every day)
Date:	Wed, 31 Oct 2007 20:30:44 +0530

Hi Raghavendra,

I would to get some tips and tricks regarding in which direction Ineed to explore.

Let me brief by setup.

I have two web servers as glusterfs client and both of them areaccessing the Glusterfs server for file storage.


volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume

The glusterfs servers and clients are identical in hardware side.

The servers are using Ultra SCSI HDD and the storage capacity is around 540GB

What are the things which I need to consider for performance improvement.

See the time difference in accessing the files...

address@hidden /storage/rvdoc# time ls -l | wc -l (From gluster client)
96514

real    11m51.644s
user    0m1.994s
sys     0m3.858s


address@hidden /storage/rvdoc# time ls -l | wc -l (From gluster server)
96514

real    0m30.812s
user    0m1.542s
sys     0m2.442s


If you want any more info from my side, please inform...

Thanks,
Joseph




At 08:38 PM 10/30/2007, Joseph Job wrote:

Hi Raghavendra,

Thanks for your help, I would like to know the default value, howmuch seconds it is ?


Thanks,
Joseph

At 05:07 PM 10/30/2007, Raghavendra G wrote:

Hi Joseph,

transport-timeout in clients is very less in your configuration (4seconds). Use a higher timeout or just comment out the option touse the default one, which should be sufficient.


regards,

On Oct 30, 2007 3:16 PM, Joseph Job<<mailto:address@hidden>address@hidden> wrote:

My Setup details..
Operating System : Trustix Secure Linux release 3.0.5 (Mirch Masala)
Kernel Version : 2.6.19.7-1
Gluster version :
glusterfs-server-1.3.6-1
glusterfs-client-1.3.6-1
glusterfs-common-1.3.6-1
glusterfs-devel-1.3.6-1
Working Mode.
Two web servers are there and both web servers are using Gluster file
server in TCP network for file access.
see my server side configuration...


Server 1
## Define the stroage
volume fs3-storage
 type storage/posix                   # POSIX FS translator
 option directory /storage            # Export this directory
end-volume

volume iothreads #iothreads can giveperformance a boost

  type performance/io-threads
  option thread-count 16
  subvolumes fs3-storage
end-volume
## Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
 option listen-port 6996              # Default is 6996
 option client-volume-filename /var/log/glusterfs/client.vol
 subvolumes iothreads
 option auth.ip.iothreads.allow * # Allow access to "brick" volume
end-volume
Server 2
## Define the stroage
volume fs4-storage
 type storage/posix                   # POSIX FS translator
 option directory /storage            # Export this directory
end-volume

volume iothreads #iothreads can giveperformance a boost

  type performance/io-threads
  option thread-count 16
  subvolumes fs4-storage
end-volume
## Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
 option listen-port 6996              # Default is 6996
 option client-volume-filename /var/log/glusterfs/client.vol
 subvolumes iothreads
 option auth.ip.iothreads.allow * # Allow access to "brick" volume
end-volume

Client side configuration...
Client 1
### Add client feature and attach to remote subvolume
volume client_214
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport

option remote-host <http://10.10.0.214>10.10.0.214 # IPaddress of the remote brick

 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume
### Add client feature and attach to remote subvolume
volume client_213
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport

option remote-host <http://10.10.0.213>10.10.0.213 # IPaddress of the remote brick

 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume
volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume
volume iothreads    #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 8
  subvolumes afrbricks
end-volume
##########################
Client 2
Client 1
### Add client feature and attach to remote subvolume
volume client_214
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport

option remote-host <http://10.10.0.214>10.10.0.214 # IPaddress of the remote brick

 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume
### Add client feature and attach to remote subvolume
volume client_213
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport

option remote-host <http://10.10.0.213>10.10.0.213 # IPaddress of the remote brick

 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume
volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume
volume iothreads    #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 8
  subvolumes afrbricks
end-volume
##########################

I am mounting the server to client with glusterfs -f
/etc/glusterfs/glusterfs- client.vol /storage/
I could able to mount, files are getting replicated to both file
servers. But the problem, the mounting is braking very frequently...
I am getting the error in glusterfs.log
2007-10-30 00:46:10 C [ tcp.c:81:tcp_disconnect] client_213:
connection disconnected
2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_214:
bailing transport
2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_213:
bailing transport
2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_214:
connection disconnected
2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_213:
connection disconnected
But the physical connection is still there...I can ping from client
to servers...
The server is using gigabit networking bonding.
I am using kernel with Fuse supported
address@hidden ~# lsmod
Module                  Size  Used by
fuse                   39444  2
ipv6                  221344  24
tg3                   105860  0
bonding                79224  0
jfs                   163564  2
usbhid                 35936  0
ohci_hcd               18564  0
usbcore               112772  3 usbhid,ohci_hcd
parport_pc             21956  0
parport                20032  1 parport_pc
shpchp                 32416  0
serverworks             8840  0 [permanent]
cciss                  54020  8
dm_mod                 49432  0
sd_mod                 17024  0
piix                    9604  0 [permanent]
ide_disk               14336  0
ide_generic             2048  0 [permanent]
ide_core              106444  4 serverworks,piix,ide_disk,ide_generic

Also see the glusterfsd.log in server side
2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected