Re: [Gluster-devel] successive bonnie++ tests taking longer and longer to run (system load steadily increasing)

[Raghavendra G]

Hi Daniel,

We ran some performance tests in our lab, with reasonable cache-size for io-cache (so that it won't consume all the memory present in the system). 

setup: A simple client/server configuration with/without io-cache on client side.

====

Here are the results.

without io-cache:
=============
address@hidden test]# time bonnie++ -u anush
Using uid:1013, gid:1013.
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
client05         4G 24972  36 28187   1 13149   1 26898  40 30860   1 234.5   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1226   2  4956   2  1806   2  1201   3  5522   3  1874   3

with io-cache:
===========
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
client05         4G 24920  37 28529   1  4576   0 36690  52 61428   3 186.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1230   2  4945   3  1816   3  1134   2  5576   5  1906   3

As you can see the test which is most affected by the presence of io-cache is Rewrite. During rewrite, bonnie++ reads an 8kb block, dirties it and writes back to same location in file. This test consists of 3 system calls - read, lseek and write. Since cache is flushed at each write, effectively data is always read from backend filesystem. However the difference is that, since io-cache stores data in pages of 128KB each, it also does reads in terms of 128KB block sizes. This it does in the hope that the overhead involved in reading 128KB blocks (instead of in block-sizes what fuse has asked) is acceptable, since successive reads can be fetched from cache. Hence every read of 8KB block size from bonnie++ is transformed into a read of 128KB block-size is io-cache is present. This is the cause of lower performance when io-cache is present during rewrites.

To confirm what we had suspected, we changed the page-size in io-cache to 8KB and following are the results.
without io-cache:
=============
address@hidden test]# bonnie++ -u anush
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-

                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
client05         4G 24917  35 28243   1 13130   1 27295  39 30836   1 239.8   0

                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  1224   2  4949   2  1798   2  1186   2  5509   3  1876   2

with io-cache:
==========
address@hidden test]#  bonnie++ -u anush                                                                           
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
client05         4G 24925  36 28019   2 12786   1 31224  44 28096   1 220.8   0

                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16  1192   3  4897   2  1784   3  1215   3  5477   4  1842   3

As we can see, there is not much difference in Rewrite performance with/without io-cache. Hence if there are writes to the same offsets from which reads have been done, data will always be read from back-end and overhead of io-cache can be significant. In these scenarios io-cache is not recommended. However for other cases, such as mostly read-only operations, performance boost from io-cache can be significant.

regards,

On Tue, Mar 2, 2010 at 1:30 PM, Daniel Maher <address@hidden> wrote:

Daniel Maher wrote:

Raghavendra G wrote:

   However, at two points during the multi-day test run, something
   strange happened.  The time to completion dropped _dramatically_,
   and stayed there for numerous iterations, before jumping back up again :


Mostly reads are being served from io-cache?

Perhaps ; it is worth noting that even though the operations are consistent, the data are being generated randomly.  I concede that, statistically speaking, some of those 0's and 1's would be cached effectively, but this shouldn't account for a sudden ~ 50% increase in efficiency that, just as suddenly as it appears, disappears again.

While it is irresponsible to extrapolate based on three points, my newest test run with io-cache disabled has yielded 10m30s, 10m36s, and 10m34s so far...

After hundreds of iterations the average « real » time per run was 10m25.522s .  This was with io-cache totally disabled.

Thus, it has been shown that given a series of systematic read and write operations on progressively larger files filled with random data, the usage of io-cache is not appropriate (and will cause severe performance problems).

Of course, one could have postulated this intuitively - but there's nothing like some hard data to back up a hypothesis. :)

The real mystery is why the test with a small io-cache yielded two groups of highly varient TTCs...

--
Daniel Maher <dma+gluster AT witbe DOT net>


_______________________________________________
Gluster-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel

-- 
Raghavendra G