[Gluster-devel] Seeing non-priv port + auth issue in the gluster brick log

[Deepak C Shetty]

Hi All,
   I am trying to mount a gluster volume from inside a VM (being used 
as a VDSM host) and seeing the below issue.
From the VM, I am able to ping, telnet to the gluster host (no 
networking issues present)

*Client side*
============

address@hidden glusterfs]# mount -t glusterfs 9.121.60.166:dpkvol /mnt
Mount failed. Please check the log file for more details.

(Using IP or hostname in the mount cmdline doesn't change anything)

mnt.log
-------

[2013-05-11 06:38:54.199518] I [glusterfsd.c:1878:main] 
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 
3.4.0beta1 (/usr/sbin/glusterfs --volfile-id=dpkvol 
--volfile-server=9.121.60.166 /mnt)
[2013-05-11 06:38:54.203474] I [socket.c:3480:socket_init] 0-glusterfs: 
SSL support is NOT enabled
[2013-05-11 06:38:54.203652] I [socket.c:3495:socket_init] 0-glusterfs: 
using system polling thread
[2013-05-11 06:38:54.206592] W [common-utils.c:2330:gf_ports_reserved] 
0-glusterfs-socket:  is not a valid port identifier
[2013-05-11 06:38:55.253221] I [socket.c:3480:socket_init] 
0-dpkvol-client-0: SSL support is NOT enabled
[2013-05-11 06:38:55.253268] I [socket.c:3495:socket_init] 
0-dpkvol-client-0: using system polling thread
[2013-05-11 06:38:55.253300] I [client.c:2154:notify] 0-dpkvol-client-0: 
parent translators are ready, attempting connect on transport
[2013-05-11 06:38:55.255834] W [common-utils.c:2330:gf_ports_reserved] 
0-glusterfs-socket:  is not a valid port identifier
Given volfile:
+------------------------------------------------------------------------------+
  1: volume dpkvol-client-0
  2:     type protocol/client
  3:     option transport-type tcp
  4:     option remote-subvolume /home/dpkshetty/brick
  5:     option remote-host llmvm03
  6: end-volume
  7:
  8: volume dpkvol-dht
  9:     type cluster/distribute
 10:     subvolumes dpkvol-client-0
 11: end-volume
 12:
 13: volume dpkvol-write-behind
 14:     type performance/write-behind
 15:     subvolumes dpkvol-dht
 16: end-volume
 17:
 18: volume dpkvol-read-ahead
 19:     type performance/read-ahead
 20:     subvolumes dpkvol-write-behind
 21: end-volume
 22:
 23: volume dpkvol-io-cache
 24:     type performance/io-cache
 25:     subvolumes dpkvol-read-ahead
 26: end-volume
 27:
 28: volume dpkvol-quick-read
 29:     type performance/quick-read
 30:     subvolumes dpkvol-io-cache
 31: end-volume
 32:
 33: volume dpkvol-open-behind
 34:     type performance/open-behind
 35:     subvolumes dpkvol-quick-read
 36: end-volume
 37:
 38: volume dpkvol-md-cache
 39:     type performance/md-cache
 40:     subvolumes dpkvol-open-behind
 41: end-volume
 42:
 43: volume dpkvol
 44:     type debug/io-stats
 45:     option count-fop-hits off
 46:     option latency-measurement off
 47:     subvolumes dpkvol-md-cache
 48: end-volume

+------------------------------------------------------------------------------+
[2013-05-11 06:38:57.007678] I [rpc-clnt.c:1670:rpc_clnt_reconfig] 
0-dpkvol-client-0: changing port to 49152 (from 0)
[2013-05-11 06:38:57.007973] W [socket.c:514:__socket_rwv] 
0-dpkvol-client-0: readv failed (No data available)
[2013-05-11 06:38:57.020391] W [common-utils.c:2330:gf_ports_reserved] 
0-glusterfs-socket:  is not a valid port identifier
[2013-05-11 06:38:58.375306] I 
[client-handshake.c:1658:select_server_supported_programs] 
0-dpkvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2013-05-11 06:38:59.077357] W 
[client-handshake.c:1365:client_setvolume_cbk] 0-dpkvol-client-0: failed 
to set the volume (Permission denied)
[2013-05-11 06:38:59.077535] W 
[client-handshake.c:1391:client_setvolume_cbk] 0-dpkvol-client-0: failed 
to get 'process-uuid' from reply dict
[2013-05-11 06:38:59.077571] E 
[client-handshake.c:1397:client_setvolume_cbk] 0-dpkvol-client-0: 
SETVOLUME on remote-host failed: Authentication failed
[2013-05-11 06:38:59.077606] I 
[client-handshake.c:1482:client_setvolume_cbk] 0-dpkvol-client-0: 
sending AUTH_FAILED event
[2013-05-11 06:38:59.077647] E [fuse-bridge.c:4788:notify] 0-fuse: 
Server authenication failed. Shutting down.
[2013-05-11 06:38:59.077680] I [fuse-bridge.c:5212:fini] 0-fuse: 
Unmounting '/mnt'.
[2013-05-11 06:38:59.082462] W [glusterfsd.c:970:cleanup_and_exit] 
(-->/usr/lib64/libc.so.6(clone+0x6d) [0x3cbd0f199d] 
(-->/usr/lib64/libpthread.so.0() [0x3cbd407d14] 
(-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x406be5]))) 0-: 
received signum (15), shutting down

version
---------

address@hidden glusterfs]# gluster --version
glusterfs 3.4.0beta1 built on May 10 2013 17:55:27
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU 
General Public License.



*Server side* (gluster host)
=============

brick log
----------

[2013-05-11 06:40:19.912512] E [addr.c:152:gf_auth] 0-auth/addr: client 
is bound to port 1070 which is not privileged
[2013-05-11 06:40:19.912610] E [authenticate.c:246:gf_authenticate] 
0-auth: no authentication module is interested in accepting 
remote-client (null)
[2013-05-11 06:40:19.912639] E [server-handshake.c:587:server_setvolume] 
0-dpkvol-server: Cannot authenticate client from 
vdsm_tsm_int-7221-2013/05/11-06:38:54:195128-dpkvol-client-0-0 3.4.0beta1
[2013-05-11 06:40:20.611853] I [server.c:771:server_rpc_notify] 
0-dpkvol-server: disconnecting connectionfrom 
vdsm_tsm_int-7221-2013/05/11-06:38:54:195128-dpkvol-client-0-0
[2013-05-11 06:40:20.611908] I 
[server-helpers.c:735:server_connection_put] 0-dpkvol-server: Shutting 
down connection 
vdsm_tsm_int-7221-2013/05/11-06:38:54:195128-dpkvol-client-0-0
[2013-05-11 06:40:20.611937] I 
[server-helpers.c:623:server_connection_destroy] 0-dpkvol-server: 
destroyed connection of 
vdsm_tsm_int-7221-2013/05/11-06:38:54:195128-dpkvol-client-0-0

glusterd.vol
------------

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option rpc-auth-allow-insecure on
end-volume

brick vol
----------

(snip..)
volume dpkvol-server
    type protocol/server
    option rpc-auth-allow-insecure on
    option auth.addr./home/dpkshetty/brick.allow null
    option auth.login.91e851d7-2f55-4946-abc2-dd18eeba4a93.password 
6c740e73-ff98-4991-b08c-1f1e9c0f9ee9
    option auth.login./home/dpkshetty/brick.allow 
91e851d7-2f55-4946-abc2-dd18eeba4a93
    option transport-type tcp
    subvolumes /home/dpkshetty/brick
end-volume

(removing auth.login lines also doesnt have any effect)

volume info
-----------

gluster volume info

Volume Name: dpkvol
Type: Distribute
Volume ID: 71fb0238-6661-4c06-ba5c-7f36c399330c
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: llmvm03:/home/dpkshetty/brick
Options Reconfigured:
server.allow-insecure: on

version
-------

gluster --version
glusterfs 3.4.0alpha2 built on Apr 10 2013 16:21:16
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU 
General Public License.



*Questions/Observations*
========================

1) Inspite of having server.allow-insecure: on (in volume option) and 
option rpc-auth-allow-insecure on (in glusterd volfile)... why do i 
still see the non-priv port error in brick logs ?
(I am running the mount cmdline as root inside VM, but its possible QEMU 
(which hosts the VM) is translating the n/w port to non-priv, but still 
auth allow insecure should have taken care of this)

2) For the same gluster host, if i try to connect from my laptop 
(instead of VM), mount works and i don't see any errors in the brick log

3) gluster --remote-host=<server ip/host> volume info works from inside 
VM (which means rpc-auth-allow-insecure on (in glusterd volfile) is 
working for this case), but not working for the mount case

4) The auth issue (from my obs) is kicking in only when non-priv port is 
being detected on the server side.. but that should not be the case as 
insecure options are set

5) Could the version mismatch between the client server be any reason 
here ?

thanx,
deepak