[Gluster-devel] RFC on posix locks migration to new graph after a switch

[Raghavendra Gowdappa]

Avati,

We had relied on posix lock-healing (here after locks refer to posix locks) 
done by protocol/client for lock migration to new graph. Lock healing is a 
feature implemented by protocol/client which simply reacquires all the granted 
locks stored in fd context after a reconnect to server. The way we leverage 
this lock healing feature of protocol client to migrate posix locks to new 
graph is: we migrate fds to new graph by opening a new fd on the same file in 
new graph (with fd context copied from old graph) and protocol/client 
reacquires all the granted locks in fd context. But, this solution has 
following issues:

1.If we open fds in new graph even before cleaning up of the old-transport, 
lock requests sent by protocol/client as part of healing will conflict with 
locks held on old-tranport and hence will fail (Note that with only client-side 
graph switch there is a single inode on server corresponding to two inodes - 
one corresponding to each of old and new graphs - on client). As a result locks 
are not migrated. The problem could've been solved if protocol/client had 
issued SETLKW requests instead of SETLK (the lock requests issued as part of 
healing would be granted when old-transport disconnects eventually). But, that 
has different set of issues. Even then, this is not a fool-proof solution, 
since there might already be other conflicting lock requests in the lock wait 
queue when protocol/client starts lock healing resulting in failure of 
lock-heal.

2. If we open fds in new graph after cleaning of old-transport, there is a 
window of time b/w old-tranport cleanup and lock-heal in new graph where 
potentially conflicting lock requests could be granted, there by causing lock 
requests sent as part of lock healing to fail.

One solution I can think of is to bring in a SETLK_MIGRATE lock command. 
SETLK_MIGRATE takes a transport identifier as a parameter along with usual 
arguments SETLK/SETLKW take (like lock range, lock-owner etc). SETLK_MIGRATE 
command migrates a lock from the transport passed as a parameter to the 
transport on which request came in, if two locks conflict only because they 
came from two different transports (all else - lock-range, lock-owner etc - 
being same). On absence of any live locks, SETLK_MIGRATE behaves similar to 
SETLK command.

protocol/client can make use of this SETLK_MIGRATE command in lock requests it 
sends as part of lock heal during open fop to migrate locks to new graph. 
Assuming that old-transport is not cleaned up at the time of lock-heal, 
SETLK_MIGRATE atomically migrates locks from old-transport to new-transport (on 
server). Now, the difficulty is in getting the identifier to old-transport on 
server from which locks are currently held. This can be solved if we store the 
peer transport identifier in lk-context on client (which can be easily obtained 
in an lk reply). We can pass the same transport identifier to server during 
healing.

I haven't yet completely thought of some issues like whether protocol/client 
can unconditionally use SETLK_MIGRATE in all lock requests it sends as part of 
healing or it should use SETLK_MIGRATE only during first attempt of healing 
after a graph-switch. However even if protocol/client wants to make such 
distinction, it can be easily worked out (either by fuse setting a special 
"migrate" key in xdata of open calls it sends as part of fd-migration or some 
different mechanism).

Please let me know your thoughts on this.

regards,
Raghavendra.