Hi Guys,
I am using gluster 3.4.2 and doing some test with replica=4, and got following 'unable to get lock error', while it seems the lock has been acquired by itself already. is this a deadlock?
any other volume command will be fail with error message like 'another transaction is in progress'.
$ sudo gluster volume status x9db
Status of volume: testvol
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick ip1:/home/cluster/db 49152 Y 2244
Brick ip2:/home/cluster/db 49152 Y 2225
Brick ip3:/home/cluster/db 49152 Y 2302
Brick ip4:/home/cluster/db 49152 Y 2302
NFS Server on localhost 2049 Y 5411
Self-heal Daemon on localhost N/A Y 5418
NFS Server on ip2 2049 Y 2279
Self-heal Daemon ip2 N/A Y 2284
NFS Server on ip3 2049 Y 2316
Self-heal Daemon on ip3 N/A Y 2321
NFS Server on ip4 2049 Y 2316
Self-heal Daemon on ip4 N/A Y 2321
There are no active volume tasks
s$ sudo gluster volume info x9db
Volume Name: testvol
Type: Replicate
Volume ID: 6a1159aa-3116-45c3-8611-16aaab76265a
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: ip1:/home/cluster/test
Brick2: ip2:/home/cluster/test
Brick3: ip3:/home/cluster/test
Brick4: ip4:/home/cluster/test
[2014-01-25 11:12:31.711699] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2014-01-25 11:12:42.143165] I [glusterd-handler.c:3260:__glusterd_handle_status_volume] 0-management: Received status volume req for volume x9db
[2014-01-25 11:12:46.061212] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)
[2014-01-25 11:12:47.547211] I [glusterd-handshake.c:557:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
[2014-01-25 11:12:47.655510] I [glusterd-volume-ops.c:478:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume x9db
[2014-01-25 11:12:47.766356] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614
[2014-01-25 11:12:47.766386] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:47.766398] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2014-01-25 11:12:47.766410] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:12:47.766426] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108
[2014-01-25 11:12:47.848456] I [glusterd-handler.c:1956:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:47.883926] I [glusterd-handler.c:2987:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 172.16.1.105 (0), ret: 0
[2014-01-25 11:12:47.997692] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, host: 172.16.1.104
[2014-01-25 11:12:47.997715] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, host: 172.16.1.107
[2014-01-25 11:12:47.997729] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108
[2014-01-25 11:12:48.098660] I [glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614
[2014-01-25 11:12:48.098696] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6
[2014-01-25 11:12:48.098711] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:48.098718] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2014-01-25 11:12:48.098727] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108
[2014-01-25 11:12:48.098738] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:48.098764] I [glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6
[2014-01-25 11:12:49.036676] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:49.036722] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:49.036734] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2014-01-25 11:12:49.036746] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:49.036759] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:12:51.052062] I [glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108, port: 0
[2014-01-25 11:12:51.152936] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:12:51.152983] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:51.153006] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:51.153017] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2014-01-25 11:12:51.153029] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:51.153042] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:13:18.695488] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:13:18.695534] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:13:18.695547] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2014-01-25 11:13:18.695561] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:13:18.695575] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:13:46.049416] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:13:46.049481] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:13:46.049985] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:13:46.050019] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:13:46.050032] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:17:37.710091] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:17:37.710159] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:17:37.710605] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:17:37.710638] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:17:37.710651] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:18:25.706653] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:18:25.706716] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:18:25.707379] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:18:25.707413] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:18:25.707426] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
[2014-01-25 11:20:59.713622] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2014-01-25 11:21:03.963492] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:21:03.963524] E [glusterd-syncop.c:1023:gd_sync_task_begin] 0-management: Unable to acquire lock
[2014-01-25 11:22:49.047317] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(4)) xid = 0x29x sent = 2014-01-25 11:12:48.102044. timeout = 600
[2014-01-25 11:22:49.048264] E [glusterd-rpc-ops.c:664:__glusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:22:49.048287] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!
[2014-01-25 11:22:49.048299] C [glusterd-op-sm.c:3648:glusterd_op_txn_complete] 0-management: Unable to clear local lock, ret: -1
[2014-01-25 11:22:49.048350] E [glusterd-rpc-ops.c:126:glusterd_op_send_cli_response] 0-management: invalid operation
[2014-01-25 11:22:49.048392] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
[2014-01-25 11:22:49.048405] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 31) to rpc-transport (socket.management)
[2014-01-25 11:22:49.048421] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed