Re: [Gluster-devel] Re: [Gluster-users] I/O fair share to avoid I/O bottlenecks on small clsuters

[Gordan Bobic]

Jeff Darcy wrote:

> > > Unfortunately, I/O traffic shaping is still in its infancy
> > > compared to what's available for networking - or perhaps even "infancy"
> > > is too generous.  As far as the I/O stack is concerned, all of the
> > > traffic is coming from the glusterfsd process(es) without
> > > differentiation, so even if the functionality to apportion I/O amongst
> > > tasks existed it wouldn't be usable without more information.  Maybe
> > > some day...
> > I don't think this would even be useful. It sounds like seeking more 
> > finely grained (sub-process level!) control over disk I/O prioritisation 
> > without there even being a clearly presented case about the current 
> > functionality (ionice) not being sufficient.
>
> Does such a case really need to be made explicitly?  Dividing processes
> into classes is all well and good, but there can still be contention
> between processes in the same class.  Being able to resolve that
> contention in a fair and/or deterministic way is still useful, and still
> unaddressed.

Hmm, perhaps. I never found that the degree of granularity that ionice 
provides is insufficient.

> In any case, that might be a moot point.  I interpreted Ran's problem as
> VMs running on GlusterFS *clients* causing contention at the GlusterFS
> *servers*.  Maybe that was incorrect, but even if Ran doesn't face that
> problem others do.  I certainly see and hear about it a lot from where I
> sit at Red Hat, and no amount of tweaking at the hypervisor (i.e.
> GlusterFS client) level will solve it.

So the problem is that a small number of clients can overwhelm the 
servers and the other clients then get very little throughput to the 
servers? If that is the case, it would imply that there is a vastly 
different "cost" top operations in glfs, and that a few clients 
hammering the expensive operations can result in the more conventional 
workload clients slowing down. If that really is the case, then, OK, I 
agree - that is something that perhaps needs to be looked into at glfs 
protocol level.

But that is now what my interpretation of the problem was. The way I 
read it, the complaint was that glusterfsd causes a lot of disk I/O, 
which in turn makes the VM containing it eat all of the host's disk I/O 
and other VMs on that host slow down. Hence why I was suggesting 
ionice-ing the qemu process containing the server.

Gordan