El cheapo 10Gbps networking

25Feb13

I’ve been hitting the limits of gigabit ethernet at home for quite a while now, and as I spend more time working with cloud technologies this started to frustrate me.

I’d heard of other folk getting good results with second hand Infiniband cards and decided to give it a go myself.

I bought two Voltaire dual-port Infiniband adapters – a 4X SDR PCI-E x4 card. And in a 2 metre 8470 cable, and we’re in business.

There are other, more comprehensive guides around to setting this up – e.g. http://davidhunt.ie/wp/?p=2291 or http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html

On ubuntu the hardware was autodetected; all I needed to do was:

modprobe ib_ipoib
sudo apt-get install opensm # on one machine

And configure /etc/network/interfaces – e.g.:

iface ib1 inet static
address 192.168.2.3
netmask 255.255.255.0
network 192.168.2.0
up echo connected >`find /sys -name mode | grep ib1`
up echo 65520 >`find /sys -name mtu | grep ib1`

With no further tuning I was able to get 2Gbps doing linear file copies via Samba, which I suspect is rather pushing the limits of my circa 2007 home server – I’ll investigate futher to identify where the bottlenecks are, but the networking itself I suspect is ok – netperf got me 6.7Gbps in a trivial test.

About these ads


4 Responses to “El cheapo 10Gbps networking”

  1. 1 John Meinel

    What are you doing that you are hitting the limits of GigE? Testing cloud deployments and running into time streaming the images?
    Streaming multiple 4k HD video streams? :)

    • 2 rbtcollins

      Schlepping lots of machine images around mainly. The odd large (0.5TB) backup / data remirror.

  2. Unfortunately, as you are probably aware, even though it’s cheap Ethernet over Infiniband, it’s Ethernet over Infiniband. Your bottleneck is going to be your CPU first, then your Infiniband switch if you have one. Instead, you need to get RDMA working, and lower the CPU several times. Then you’ll get full 10 Gbps, with minimal effort.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 1,056 other followers

%d bloggers like this: