Interesting. I think it's possible that Infiniband will one day work its way to desktop computers, but there are a lot of players in the game that have to agree to make it work and make it popular.
Why switch to infiniband? Ethernet can go just as fast (not as good of latency) and the cables are cheap and ubiquitous. The only thing stopping higher speeds is cost. I have been watching 10GbE PCI-E cards for the last year or two and they have dramatically dropped in price ($200 from newegg). You can get server motherboards with 2 integrated 10GbE ports for $500 as well. 10GbE does run over cat6 by the way.
I think for general-purpose workloads, it's not relevant. However, some of the IB-specific capabilities like RDMA are extremely interesting for clustering use cases. You can dramatically reduce the overhead of moving data around between nodes by bypassing the kernel entirely.
I understand that sort of use case, but this article is asking about desktop computers. It almost seems like the author only sees it used in high performance situations so clearly it is better for all situations.
I don't think the use case envisioned here is for home computers with InfiniBand, it's for work stations in corporate environments. Transferring large data sets to local servers is quite common, and can take a (relatively) long time even over Gigabit Ethernet.
Corporations can afford to buy pricy network hardware for the people who need it (or they can skimp on engineering resources and congratulate themselves over and over again over how thrifty they're being).
I read desktop to mostly mean consumer, perhaps erroneously?
It's mostly for moving things around your internal network. I would love to have 10Gb to my NAS, because then my backups would take seconds instead of minutes. (First world problems, I know).
In visual effects, there is need to transfer around massive data sets. WE have found that 1 Gbps Ethernet just doens't cut it, but right now 10 Gbps Ethernet solution are just too expensive for now, or require specialized cables (fiber optics).
I would really like those prices to come down and also to use just regular CAT cables.
Right now it seems that the switch to faster than 1 Gbps networking is just too costly and not easily incremental (because of the switch to optical generally.)
10GBASE-T isn't popular because your cable runs are cut short.
Costs have come down a lot on fiber. A short range SFP is like $120 or so, and short range cables are like $3 per meter.
What a lot of people do as a stopgap is binding ports together, using something like LACP. There's trade-offs of course. A 4 port 1000BASE-T card could be used to boost a few important servers.
10GBASE-T is pretty well established for top-of-rack datacenter switches. Intel's X520 chipset is pretty common and any major switch vendor will happily sell you a 24x or 48x 10-gig switch (HP's JG336A for example).
Twinax is arguably better if you're building a rack from scratch, they're lower power, but if you're upgrading something existing it's hard to justify rewiring everything. Also twinax cables are expensive and annoying to work with.
10GBaseT is cheap. Look at Arista networks. This idea is just not well reasoned. Ethernet had all the "stuff" needed to make it work when you plug it in. IB requires you to do a lot of configuration on the IB switch side, etc.
Would you want every workstation to have direct access to the cluster - ok I get for loading data it might be useful but IN that case surely you would have special purpose nodes for that
There is no practical or commercial motivation for IB on consumer devices. Sure one may make a case that a home server or a media center can use IB to stream photos and movies to other devices. At this point, we have to stop and think like an average consumer and not an IT professional. And an average consumer prefers no wires forget running copper or fiber. And what benefit would it offer over wireless even if they were to run the cable? None from the consumer perspective. You can stream HD over wireless without any problems.
Sure, we all prefer wired over wireless when available. But not in home, and not by an average consumer.
Intel will integrate IB with their future processors, but they will not be consumer based SKUs.
There are niches -- some people (interested parties for sure) even advocate for fiber to the desktop as a cost savings: http://www.thefoa.org/tech/allfiber.htm
But for regular consumers, wireless is the driver. Power users will care about cat 6. The rest is just niche.
You can do InfiniBand on the desktop, but it necessarily doesn't give the performance you might be expecting. I think I read about this on HN originally...
Edit: From the edit on the follow-up post about getting it working on Ubuntu:
>Note: iperf maxed out at 1.2 gbps, and on the current linux install, I couldnt get netperf client working at all. netserver would work, but only showed a throughput of 25mbps from a Win7 client. HOWEVER, when I set up the raid with 6 old 160G drives, the “hdparm -t /dev/md0p1″ showed 250MB/sec reads, and I got the same from the Win7 machine using samba across the infiniband fabric. This seems to indicate that iperf and netperf are completely unreliable for testing this type of connection. Bear in mind though that I did have netperf running on the previous ubuntu install, but that installation was so messy I don’t know what drivers and user-space software was running. I reckon it’s the kind of think that may be fixed in the stock Ubuntu install in the near future. For the moment, just go with real-world testing, i.e. copying large files from ramdisk to ramdisk, for example.
23 comments
[ 3.3 ms ] story [ 56.9 ms ] threadEven in Germany, where the author lives, the top bandwidth appears to be about 50 Mb/s:
http://www.pcmag.com/article2/0,2817,2457284,00.asp
Which although it is still much better than the states:
http://ispspeedindex.netflix.com/results/usa/graph
They're both well below 1 Gb/s, no?
I read desktop to mostly mean consumer, perhaps erroneously?
http://en.wikipedia.org/wiki/10-gigabit_Ethernet
I would really like those prices to come down and also to use just regular CAT cables.
Right now it seems that the switch to faster than 1 Gbps networking is just too costly and not easily incremental (because of the switch to optical generally.)
PS. There is 10GBASE-T but I think there is zero adoption or promotion of this substandard: http://en.wikipedia.org/wiki/10-gigabit_Ethernet#10GBASE-T
Costs have come down a lot on fiber. A short range SFP is like $120 or so, and short range cables are like $3 per meter.
What a lot of people do as a stopgap is binding ports together, using something like LACP. There's trade-offs of course. A 4 port 1000BASE-T card could be used to boost a few important servers.
Twinax is arguably better if you're building a rack from scratch, they're lower power, but if you're upgrading something existing it's hard to justify rewiring everything. Also twinax cables are expensive and annoying to work with.
Sure, we all prefer wired over wireless when available. But not in home, and not by an average consumer.
Intel will integrate IB with their future processors, but they will not be consumer based SKUs.
But for regular consumers, wireless is the driver. Power users will care about cat 6. The rest is just niche.
http://www.davidhunt.ie/infiniband-at-home-10gb-networking-o...
Edit: From the edit on the follow-up post about getting it working on Ubuntu:
>Note: iperf maxed out at 1.2 gbps, and on the current linux install, I couldnt get netperf client working at all. netserver would work, but only showed a throughput of 25mbps from a Win7 client. HOWEVER, when I set up the raid with 6 old 160G drives, the “hdparm -t /dev/md0p1″ showed 250MB/sec reads, and I got the same from the Win7 machine using samba across the infiniband fabric. This seems to indicate that iperf and netperf are completely unreliable for testing this type of connection. Bear in mind though that I did have netperf running on the previous ubuntu install, but that installation was so messy I don’t know what drivers and user-space software was running. I reckon it’s the kind of think that may be fixed in the stock Ubuntu install in the near future. For the moment, just go with real-world testing, i.e. copying large files from ramdisk to ramdisk, for example.