Re: [Gluster-devel] Gluster CLI for setting options for glusterd.vol

[Deepak C Shetty]

On 06/27/2013 12:34 PM, Kanagaraj M wrote:

On Thu, Jun 27, 2013 at 11:20 AM, Deepak C Shetty <address@hidden> wrote:

On 06/24/2013 06:21 PM, Vijay Bellur wrote:

On 06/20/2013 07:28 PM, M. Mohan Kumar wrote:

Vijay Bellur <address@hidden> writes:

On 06/19/2013 09:51 PM, M. Mohan Kumar wrote:

Hello,

When qemu is invoked by a non-root user with -drive
file=gluster://server/volname/imagename option, unprivileged port is
used for gluster rpc and by default glusterd and gluster brick process
deny the request if the request is from a unprivileged port. The option
"rpc-auth-allow-insecure" needs to be enabled in glusterd.vol so that
non privileged ports can be used to access Gluster volumes.

In a typical environment VDSM might want to enable rpc-auth-allow-insecure
option and the administrator has to edit the glusterd.vol manually and
restart glusterd process.

CLI options available to enable volume specific options to work with
unprivileged ports by using gluster volume set <volname> <option>
<value>. For example per volume server.allow-insecure option can be
enabled so that unprivileged users can mount a GlusterFS volume.

But as of now there is no CLI option available to set glusterd.vol
options. How about adding a gluster CLI set option to configure
glusterd.vol options? Can following CLI command line 'gluster volume set
all <glusterd.option> <value>" be used for setting glusterd options?
IIUC "all" is a reserved volume name and we can use this reserved name
for setting glusterd option.

'volume set all' is mostly used for options that are applicable to all
volumes. Since glusterd options are beyond the scope of a volume, tying
them to the peer entity might be a good idea. We can introduce 'peer set
all <key> <value>' which sets a particular option on all peers.

You mean by 'gluster peer set all rpc-auth-allow-insecure on' will
enable insecured port access to all glusterds in the peer environment?

Yes.

This still doesn't help the VDSM usecase, when VDSM host ( aka hypervisor host ) is not part of gluster peer.
One of the goal here was to provide a cli way to modify glusterd options so that VDSM can exploit it, when Gluster volume is used as a storage domain, and VDSM needs rpc-auth-allow-insecure to be ON as VMs accessing Gluster volume natively via libgfapi will be running as non-root.

On the same lines.. how does oVirt Engine 'Volumes' GUI manage Gluster volumes.... when the oVirt host is not part of the Gluster peer ? Just wondering....

Why oVirt Engine host needs to be a gluster peer to be able to communicate with gluster(through vdsm)? As super-vdsm is running as root in the gluster node and it should be able to communicate with underlying gluster cli and can respond to oVirt engine.

Right, I forgot the fact that gluster clis are executed thru super-vdsm, so non-priv / insecure issue won't be present. But the VDSM usecase still is a issue... when VDSM host (hypervisor node) is not a gluster peer, we need a way to set the glusterd option... currently one has to do that manually, which is not a good choice.

If not, are you talking about the scenario where gluster node doesn't have vdsm installed also the storage cluster is not managed through oVirt.

So, as i see it, there are couple of scenarios.... from VDSM perspective...

For the sake of discussion, lets say VDSM (aka hypervisor host) is hostA & Gluster node/host is hostB.

Case 1: hostA is not a gluster peer. hostB is a gluster peer (obviously) , part of Gluster volume which is managed by oVirt Engine.

    hostA will access Gluster volume as non-root (as part of VM running on Gluster storage domain, using libgfapi). Since hostA is not a gluster peer but since the Gluster volume is managed by oVirt, it should be possible for the engine to first execute 'gluster peer set all rpc-auth-allow-insecure on the gluster volume, then ask VDSM to create the gluster storage domain.  From the perspective of "User friendliness", its better to have User click on something like "Allow insecure access" or "Make this gluster volume work with VDSM Gluster storage domain" which will execute 'gluster peer set all ..insecure.. ' option, so that User is aware of enabling the insecure option, instead of engine doing it w/o User's knowledge as part of gluster storage domain flow.

This should work and for this to happen seamless, the CLI support for setting glusterd is needed, so this usecase is covered.

Case 2: hostA is not a gluster peer. hostB is a gluster peer (obviously), but the Gluster volume is not managed by oVirt.

    The same as Case 1 applies, with the exception that, since the Gluster volume is not managed by oVirt, engine cannot do 'gluster peer set all rpc-auth-allow-insecure'. Ther are again 2 sub-cases here ...

    2a)  Have the user import the gluster volume into oVirt, post which it becomes like Case 1... problem solved.

    2b) For some reason, user doesn't want to import the gluster volume into oVirt.. in which case, the only way is for someone to _manually_ do 'gluster peer set all rpc-auth-allow-insecure' from one of the gluster peers, which is not a good thing since it involved manual intervention. One can argue that hostA can use --remote-host option and do 'gluster peer set all...', which will work today, but that would be insecure way of doing it, since you are allowing a non-peer to set insecure option for all peers. Also Mohan's another patch (http://review.gluster.org/#/c/5092/) adds support for IP based access control, which also disables any 'set' kind of actions using --remote-host, so --remote-host anyways won't be possible in VDSM once that patch is thru. Net, there is no way in VDSM to enable glusterd insecure option for this usecase.

Case 3: hostA is a gluster peer. hostB is a gluster peer (obviously) , part of Gluster volume which is managed by oVirt Engine.

    Since oVirt is managing is gluster volume, the same as Case 1 applies

Case 4: hostA is a gluster peer. hostB is a gluster peer (obviously) , but the Gluster volume is not managed by oVirt.

    (Someone manually set up the Gluster volume, using hypervisor hosts as peers, for using the un-used disk space on hyp host for storage)
    This is similar to Case 2, but since hostA and hostB are part of gluster peer, VDSM can do 'gluster peer set all ...insecure..' before creating storage domain. Problem happens, when new hosts are added to the datacentre which are not part of gluster peer. Note in a virt env. hosts can be dynamically added / remove. So when a VM backed by gluster storage domain migrates to a host that is not a gluster peer, this usecase is broken like in Case 2b

To Summarize:

1) 'gluster peer set all ...insecure...' cli option will definitely help and it will help more if the oVirt Volumes tab provides an UI way of invoking this on User request. This will cover some usecases as desc above

2) Case 2b & 4 - how to handle this scenario ?

thanx,
deepak

Thanks,

Kanagaraj

 

thanx,
deepak

IIUC glusterd.info file can be used to store about these parameters
similar to how volume specific options are stored in vols/<volname>/info
file?

We can persist this in glusterd.vol referred by the respective glusterd
instance.

So glusterd.vol is not [re]generated during glusterd init?

No, glusterd.vol does not get altered during init.

-Vijay

-Vijay

_______________________________________________
Gluster-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/gluster-devel