El cheapo 10Gbps networking
I’ve been hitting the limits of gigabit ethernet at home for quite a while now, and as I spend more time working with cloud technologies this started to frustrate me.
I’d heard of other folk getting good results with second hand Infiniband cards and decided to give it a go myself.
I bought two Voltaire dual-port Infiniband adapters – a 4X SDR PCI-E x4 card. And in a 2 metre 8470 cable, and we’re in business.
There are other, more comprehensive guides around to setting this up – e.g. http://davidhunt.ie/wp/?p=2291 or http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html
On ubuntu the hardware was autodetected; all I needed to do was:
modprobe ib_ipoib sudo apt-get install opensm # on one machine
And configure /etc/network/interfaces – e.g.:
iface ib1 inet static address 192.168.2.3 netmask 255.255.255.0 network 192.168.2.0 up echo connected >`find /sys -name mode | grep ib1` up echo 65520 >`find /sys -name mtu | grep ib1`
With no further tuning I was able to get 2Gbps doing linear file copies via Samba, which I suspect is rather pushing the limits of my circa 2007 home server – I’ll investigate futher to identify where the bottlenecks are, but the networking itself I suspect is ok – netperf got me 6.7Gbps in a trivial test.
Filed under: Uncategorized | 4 Comments
Tags: cloud, infiniband, performance, ubuntu
What are you doing that you are hitting the limits of GigE? Testing cloud deployments and running into time streaming the images?
Streaming multiple 4k HD video streams?
Schlepping lots of machine images around mainly. The odd large (0.5TB) backup / data remirror.
Unfortunately, as you are probably aware, even though it’s cheap Ethernet over Infiniband, it’s Ethernet over Infiniband. Your bottleneck is going to be your CPU first, then your Infiniband switch if you have one. Instead, you need to get RDMA working, and lower the CPU several times. Then you’ll get full 10 Gbps, with minimal effort.
Sure; like I said, cheap
. I’ll note though that ethernet over IB is different to IP over IB 
– https://tools.ietf.org/html/rfc4391