[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Gluster-devel] Infiniband throughput
From: |
Nathan Allen Stratton |
Subject: |
[Gluster-devel] Infiniband throughput |
Date: |
Sat, 20 Oct 2007 20:14:49 -0400 (EDT) |
Anyone know idea why the wiki does not have the config files used for
client and server when benchmarks are run?
I am only getting 170 MB/s over 2 CPU quad 2.33 GHz Zeon boxes with no
load. My local storage in each box is RAID 6 over 8 disks on 3ware 9650SE
cards (PCI express). Connectivity is single port 4x Mellanox MT23108 PCIX
cards. If I write to the RAID card I get 239 MB/s.
My Configs:
http://share.robotics.net/server_vs0.vol
http://share.robotics.net/server_vs1.vol
http://share.robotics.net/server_vs2.vol
http://share.robotics.net/client.vol (same on every box)
Write to gluster share:
address@hidden time dd if=/dev/zero of=./8gbfile bs=512k
count=16384
16384+0 records in
16384+0 records out
8589934592 bytes (8.6 GB) copied, 50.5589 seconds, 170 MB/s
real 0m50.837s
user 0m0.012s
sys 0m22.341s
Write to RAID directory:
address@hidden time dd if=/dev/zero of=./8gbfile bs=512k
count=16384
16384+0 records in
16384+0 records out
8589934592 bytes (8.6 GB) copied, 35.8675 seconds, 239 MB/s
real 0m35.893s
user 0m0.024s
sys 0m20.789s
At first I thought it may be because my Infiniband cards are PCIX vs PCIe,
but when I test that I get 477 MB/s:
address@hidden ib_rdma_bw 192.168.0.12
12415: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 |
duplex=0 | cma=0 |
12415: Local address: LID 0x03, QPN 0x10408, PSN 0x689d5f RKey 0x5f4c00bd
VAddr 0x002aaaab2f5000
12415: Remote address: LID 0x04, QPN 0x5f040a, PSN 0xce42f6, RKey
0x14fa00c0 VAddr 0x002aaaab2f5000
12415: Bandwidth peak (#0 to #954): 477.292 MB/sec
12415: Bandwidth average: 477.2 MB/sec
12415: Service Demand peak (#0 to #954): 4774 cycles/KB
12415: Service Demand Avg : 4775 cycles/KB
-Nathan
- [Gluster-devel] Infiniband throughput,
Nathan Allen Stratton <=