[Gluster-devel] Raw disk / NFS/ Glusterfs (different configs) benchmarks.

[Kevan Benson]

This is probably less than worthless, but I figured some of you might get a 
kick out of it.  It's not pretty, but it's data.  Only includes values for 
file creation and deletion, no copying or moving.  I'll probably post this in 
a while with all the configs and the source of makefiles.pl (not that it's 
anything spectacular).  If you want to see them before then, just ask.

At a later date I'll probably either thread the file creation script or use a 
more standard benchmark, and take the average of multiple instead of single 
run values.

Test Notes:
Extremely raw, tests ran only once per defined type (no averaging).
All GLFS tests were fully redundant across two servers.
Same hardware in every test
Bare disk utilized disk on client, not server
No io-threads, read-ahead or write-behind utilized
Testing file creation is not concurrent, so io-threads doesn't help.

Quick summary of GLFS results in TIME(% over best case for test) format:
Type 4000 1k files 400 10k files 40 100k files 100 1MB files    Avg time 
Client 51.779(+53%) 7.247(+31%) 2.695(+42%) 55.966(+43%) = +42.25%
Clnt/Srvr 47.045(+25%) 5.802(+5%) 1.897(+0%) 39.177(+0%) = +7.5%
Server 37.850(+0%) 5.530(+0%) 2.239(+18%) 45.250(+16%) = +8.5%

Summary Notes:
Providing redundancy by implementing AFR and unify on the client seems to 
provide the worst performance (but it might provide better fault tolerance 
and recovery, as some of my tests show)
AFR on servers and Unify on the client provides good performance, but I had 
some HA recovery problems with it I need to investigate.
AFR/Unify on the server as outlined in 
http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS
 
seems to provide almost best case performance, and I have tested HA failing 
with it quite a bit.  It works, but it's slow.
NFS gets stomped in every case, but it's not tuned at all (besides any default 
tuning RHEL/CentOS may include at the kernel level).  Particularly of note is 
the small file creation times of NFS compared to GLFS.  


####################################################
10 root directories, 10 sub directories in each root, 40 files of 1k in each 
sub directory (4000 1k files).

# Base disk
# time /root/makefiles.pl /tmp/test/
real    0m0.789s
user    0m0.308s
sys     0m0.466s
# time rm -rf /tmp/test/*
real    0m0.249s
user    0m0.010s
sys     0m0.239s

# Untuned NFS
# time /root/makefiles.pl /mnt/nfs/test/
real    3m44.871s
user    0m0.491s
sys     0m1.334s
# time rm -rf /mnt/nfs/test/*
real    1m19.988s
user    0m0.017s
sys     0m0.606s

# Untuned GLFS (client side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 
real    0m51.779s
user    0m0.324s
sys     0m0.478s
# time rm -rf /mnt/glusterfs/test/*
real    0m20.523s
user    0m0.029s
sys     0m0.274s

# Untuned GLFS (server side AFR, client side unify)
# time /root/makefiles.pl /mnt/glusterfs/test/
real    0m47.045s
user    0m0.250s
sys     0m0.543s
# time rm -rf /mnt/glusterfs/test/*
real    0m21.177s
user    0m0.009s
sys     0m0.146s

# Untuned GLFS (server side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 
real    0m37.850s
user    0m0.644s
sys     0m0.630s
# time rm -rf /mnt/glusterfs/test/*
real    0m18.106s
user    0m0.011s
sys     0m0.167s


####################################################
10 root directories, 1 sub directories in each root, 40 files of 10k in each 
sub directory. (400 10k files)

# Base disk
# time /root/makefiles.pl /tmp/test/ 10 10 1 40
real    0m0.156s
user    0m0.072s
sys     0m0.084s
# time rm -rf /tmp/test/*
real    0m0.032s
user    0m0.001s
sys     0m0.031s

# Untuned NFS
# time /root/makefiles.pl /mnt/nfs/test/ 10 10 1 40
real    0m23.532s
user    0m0.085s
sys     0m0.222s
# time rm -rf /mnt/nfs/test/*
real    0m7.685s
user    0m0.002s
sys     0m0.068s

# Untuned GLFS (client side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40
real    0m7.247s
user    0m0.075s
sys     0m0.085s
# time rm -rf /mnt/glusterfs/test/*
real    0m1.787s
user    0m0.001s
sys     0m0.021s

# Untuned GLFS (server side AFR, client side unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40
real    0m5.802s
user    0m0.080s
sys     0m0.083s
# time rm -rf /mnt/glusterfs/test/*
real    0m2.142s
user    0m0.003s
sys     0m0.021s

# Untuned GLFS (server side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 10 10 1 40
real    0m5.530s
user    0m0.065s
sys     0m0.068s
# time rm -rf /mnt/glusterfs/test/*
real    0m1.760s
user    0m0.000s
sys     0m0.026s


####################################################
1 root directories, 1 sub directories in each root, 40 files of 100k in each 
sub directory. (40 100k files)

# Base disk
# time /root/makefiles.pl /tmp/test/ 100 1 1 40
real    0m0.098s
user    0m0.052s
sys     0m0.046s
# time rm -rf /tmp/test/*
real    0m0.011s
user    0m0.001s
sys     0m0.010s

# Untuned NFS
# time /root/makefiles.pl /mnt/nfs/test/ 100 1 1 40
real    0m3.101s
user    0m0.051s
sys     0m0.083s
# time rm -rf /mnt/nfs/test/*
real    0m0.736s
user    0m0.001s
sys     0m0.014s

# Untuned GLFS (client side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40
real    0m2.695s
user    0m0.043s
sys     0m0.025s
# time rm -rf /mnt/glusterfs/test/*
real    0m0.190s
user    0m0.000s
sys     0m0.006s

# Untuned GLFS (server side AFR, client side unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40
real    0m1.897s
user    0m0.051s
sys     0m0.036s
# time rm -rf /mnt/glusterfs/test/*
real    0m0.229s
user    0m0.001s
sys     0m0.005s

# Untuned GLFS (server side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 100 1 1 40
real    0m2.239s
user    0m0.051s
sys     0m0.030s
# time rm -rf /mnt/glusterfs/test/*
real    0m0.185s
user    0m0.002s
sys     0m0.003s

####################################################
10 root directories, 1 sub directories in each root, 10 files of 1000k in each 
sub directory. (100 1 MB files)

# Base disk
# time /root/makefiles.pl /tmp/test/ 1000 10 1 10
real    0m1.653s
user    0m0.749s
sys     0m0.890s
# time rm -rf /tmp/test/*
real    0m15.333s
user    0m0.001s
sys     0m0.133s

# Untuned NFS
# time /root/makefiles.pl /mnt/nfs/test/ 1000 10 1 10
real    0m19.327s
user    0m0.672s
sys     0m0.678s
# time rm -rf /mnt/nfs/test/*
real    0m2.438s
user    0m0.002s
sys     0m0.058s

# Untuned GLFS (client side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10
real    0m55.966s
user    0m0.638s
sys     0m0.459s
# time rm -rf /mnt/glusterfs/test/*
real    0m0.671s
user    0m0.002s
sys     0m0.006s

# Untuned GLFS (server side AFR, client side unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10
real    0m39.177s
user    0m0.732s
sys     0m0.435s
# time rm -rf /mnt/glusterfs/test/*
real    0m1.088s
user    0m0.002s
sys     0m0.010s

# Untuned GLFS (server side AFR and unify)
# time /root/makefiles.pl /mnt/glusterfs/test/ 1000 10 1 10
real    0m45.250s
user    0m0.831s
sys     0m0.397s
# time rm -rf /mnt/glusterfs/test/*
real    0m0.872s
user    0m0.005s
sys     0m0.010s

-- 
- Kevan Benson
- A-1 Networks