[Gluster-devel] AFR setup with Virtual Servers crashes

[Urban Loesch]

Hi,

I'm new to this list.
First: sorry for my bad english.

I was searching for some easy and transparent Clusterfilesystem with 
failover feature and I found on Wikipedia the GlusterFS project.
It's a nice project and tried it on my test environment. I thought when 
it works good I use it in production too.

A very nice feature for me is the AFR setup. So I can replicate all the 
data over 2 Servers in RAID-1 Mode.
But it seems that I make something wrong, because the "glusterfsd" 
crashes on both nodes.
But let me explain form the beginning.

Here's my setup:
Hardware:
2 different servers for storage
1 server as client
On top of the server I use a virtual server setup (details 
http://linux-vserver.org).

OS:
Debian Sarge with self compiled 2.6.19.2 (uname -r 2.6.19.2-vs2.2.0) and 
latest stable virtual server patch.
glusterfs-1.3.0-pre3.tar.gz

What I'm trying to do:
- Create a AFR Mirror over the 2 Servers.
- Mount the Volume on Server 3 (Client).
- Install on the mounted volume the hole virtual Server with Apache, 
MySql and so on.
So I have a full redundant Virtual Server mirrored over two bricks .

Here my current confuguration:
- Serverconfig on Server 1 (brick)

### Export volume "brick" with the contents of "/home/export" directory.
volume brick
 type storage/posix                   # POSIX FS translator
 option directory /gluster        # Export this directory
end-volume

### File Locking
volume locks
 type features/posix-locks
 subvolumes brick
end-volume

### Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
option listen-port 6996               # Default is 6996
 subvolumes locks
 option auth.ip.locks.allow *         # access to "brick" volume
end-volume

- Serverconfig on Server 2 (brick-afr)
### Export volume "brick" with the contents of "/home/export" directory.
volume brick-afr
 type storage/posix                   # POSIX FS translator
 option directory /gluster-afr        # Export this directory
end-volume

### File Locking
volume locks-afr
 type features/posix-locks
 subvolumes brick-afr
end-volume

### Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
option listen-port 6996               # Default is 6996
 subvolumes locks-afr
 option auth.ip.locks-afr.allow *         # access to "brick" volume
end-volume

- Clientconfiguration on Server 3 (
### Add client feature and attach to remote subvolume of server1
volume brick
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.0.1      # IP address of the remote brick
 option remote-port 6996              # default server port is 6996
 option remote-subvolume locks        # name of the remote volume
end-volume

### Add client feature and attach to remote subvolume of brick1
volume brick-afr
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
 option remote-host 192.168.0.2      # IP address of the remote brick
 option remote-port 6996              # default server port is 6996
 option remote-subvolume locks-afr        # name of the remote volume
end-volume

### Add AFR feature to brick
volume afr
 type cluster/afr
 subvolumes brick brick-afr
 option replicate *:2                 # All files 2 copies (RAID-1)
end-volume

----------------------------------------------------------------------------------------------------------------------
I started the two Bricks in debug mode and it starts without problems.

- Server1
glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG
....
[May 10 11:52:11] [DEBUG/proto-srv.c:2919/init()] 
protocol/server:protocol/server xlator loaded
[May 10 11:52:11] [DEBUG/transport.c:83/transport_load()] 
libglusterfs/transport:attempt to load type tcp/server
[May 10 11:52:11] [DEBUG/transport.c:88/transport_load()] 
libglusterfs/transport:attempt to load file 
/usr/lib/glusterfs/1.3.0-pre3/transport/tcp/server.so

- Server2
glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG
....
[May 10 11:51:44] [DEBUG/proto-srv.c:2919/init()] 
protocol/server:protocol/server xlator loaded
[May 10 11:51:44] [DEBUG/transport.c:83/transport_load()] 
libglusterfs/transport:attempt to load type tcp/server
[May 10 11:51:44] [DEBUG/transport.c:88/transport_load()] 
libglusterfs/transport:attempt to load file 
/usr/lib/glusterfs/1.3.0-pre3/transport/tcp/server.so
------------------------------------------------------------------------------------------------------------------------------

So far so good.

After I mounted the volume on server 3 (client). It mounts without any 
problems.
glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG 
--spec-file=/etc/glusterfs/glusterfs-client.vol /var/lib/vservers/mastersql
...
[May 10 13:59:00] [DEBUG/client-protocol.c:2796/init()] 
protocol/client:defaulting transport-timeout to 120
[May 10 13:59:00] [DEBUG/transport.c:83/transport_load()] 
libglusterfs/transport:attempt to load type tcp/client
[May 10 13:59:00] [DEBUG/transport.c:88/transport_load()] 
libglusterfs/transport:attempt to load file 
/usr/lib/glusterfs/1.3.0-pre3/transport/tcp/client.so
[May 10 13:59:00] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 8
[May 10 13:59:00] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1022'
[May 10 13:59:00] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 8 in progress (non-blocking)
[May 10 13:59:00] [DEBUG/tcp-client.c:293/tcp_connect()] 
tcp/client:connection on 8 still in progress - try later

OK. Nice.
A short check on the client:
df -HT
Filesystem    Type     Size   Used  Avail Use% Mounted on
/dev/sda1     ext3      13G   2.6G   8.9G  23% /
tmpfs        tmpfs     1.1G      0   1.1G   0% /lib/init/rw
udev         tmpfs      11M    46k    11M   1% /dev
tmpfs        tmpfs     1.1G      0   1.1G   0% /dev/shm
glusterfs:24914
             fuse     9.9G   2.5G   6.9G  27% /var/lib/vservers/mastersql

Wow it works. Now I can add, remove or edit files and directories 
without problems. The file are written to all two bricks without 
problems. Performance is good too.

But then I tried to start my virtual Server (called mastersql).
The virtual server not starts and I get the a lot of following debug 
output on the client:

[May 10 14:04:43] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 4
[May 10 14:04:43] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[May 10 14:04:43] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 4 in progress (non-blocking)
[May 10 14:04:43] [DEBUG/tcp-client.c:293/tcp_connect()] 
tcp/client:connection on 4 still in progress - try later
[May 10 14:04:43] [ERROR/client-protocol.c:204/client_protocol_xfer()] 
protocol/client:transport_submit failed
[May 10 14:04:43] 
[DEBUG/client-protocol.c:2604/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x8076cf0
[May 10 14:04:43] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 7
[May 10 14:04:43] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1022'
[May 10 14:04:43] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 7 in progress (non-blocking)
[May 10 14:04:43] [DEBUG/tcp-client.c:293/tcp_connect()] 
tcp/client:connection on 7 still in progress - try later
[May 10 14:04:43] [ERROR/client-protocol.c:204/client_protocol_xfer()] 
protocol/client:transport_submit failed
[May 10 14:04:43] 
[DEBUG/client-protocol.c:2604/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x80762d0

The two mirrorservers are crashing with the following debug code:

[May 10 11:54:26] [DEBUG/tcp-server.c:134/tcp_server_notify()] 
tcp/server:Registering socket (5) for new transport object of 192.168.0.3
[May 10 11:55:22] [DEBUG/proto-srv.c:2418/mop_setvolume()] 
server-protocol:mop_setvolume: received port = 1022
[May 10 11:55:22] [DEBUG/proto-srv.c:2434/mop_setvolume()] 
server-protocol:mop_setvolume: IP addr = *, received ip addr = 192.168.0.3
[May 10 11:55:22] [DEBUG/proto-srv.c:2444/mop_setvolume()] 
server-protocol:mop_setvolume: accepted client from 192.168.0.3

Trying to set: READ  Is grantable: READ   Inserting: READTrying to set: 
UNLOCK  Is grantable: UNLOCK  Conflict with: READTrying to set: WRITE  
Is grantable: WRITE   Inserting: WRITETrying to set: UNLOCK  Is 
grantable: UNLOCK  Conflict with: WRITETrying to set: WRITE  Is 
grantable: WRITE   Inserting: WRITETrying to set: UNLOCK  Is grantable: 
UNLOCK  Conflict with: WRITETrying to set: WRITE  Is grantable: WRITE   
Inserting: WRITE[May 10 12:00:09] 
[CRITICAL/common-utils.c:215/gf_print_trace()] debug-backtrace:Got 
signal (11), printing backtrace
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/libglusterfs.so.0(gf_print_trace+0x2e) [0xb7f53a7e]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:[0xb7f60420]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so 
[0xb75d1192]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so 
[0xb75cded7]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/libglusterfs.so.0(transport_notify+0x1d) 
[0xb7f54ecd]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xe9) 
[0xb7f55b79]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/usr/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7f54f7d]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:glusterfsd [0x804924e]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8) 
[0xb7e17ea8]
[May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()] 
debug-backtrace:glusterfsd [0x8048c51]
Segmentation fault (core dumped)

It seems that there are come conflicts with "READ, WRITE, UNLOCK". But 
I'm not an expert on filesystems an locking features.

As you can see the filesystem is just mounted but not connected to the 
two bricks.
df -HT
Filesystem    Type     Size   Used  Avail Use% Mounted on
/dev/sda1     ext3      13G   2.6G   8.9G  23% /
tmpfs        tmpfs     1.1G      0   1.1G   0% /lib/init/rw
udev         tmpfs      11M    46k    11M   1% /dev
tmpfs        tmpfs     1.1G      0   1.1G   0% /dev/shm
df: `/var/lib/vservers/mastersql': Transport endpoint is not connected

I'm not sure if i make something wrong (configuration) or if it is a bug!
Can you experts please help me?

If you need any further information or something please let me know.

Thanks and regards
Urban