July 14, 2016 Stephen Perrenod
With the increasing adoption of scale-out architectures and cloud computing, high performance interconnect (HPI) technologies have become a more critical part of IT systems. Today, HPI represents its own market segment at the upper echelons of the networking equipment market, supporting applications requiring extremely low latency and exceptionally high bandwidth.
As big data analytics, machine learning, and business optimization applications become more prevalent, HPI technologies are of increasing importance for enterprises as well. These most demanding enterprise applications, as well as high performance computing (HPC) applications, are generally addressed with scale-out clusters based on large numbers of ‘skinny’ nodes. The requirement for large node counts places a heavy burden on inter-node communication.
We have identified three main technologies that are part of this new HPI market: Ethernet, InfiniBand, and Omni-Path Architecture, but there are also other proprietary technologies in the HPI segment, such as Cray’s “Aries” XC interconnect and SGI’s NUMALink.
Evaluation Criteria For High Performance Interconnects
Whatever technology is chosen, inter-node communication must be reliable, be of high bandwidth, and must have sufficiently low latency. Achievable performance should be high for both large and small messages.
Currently, Ethernet bandwidth peaks at 100 Gb/sec, as does InfiniBand. However, achievable latencies are another matter. Ethernet switch vendors are proud when their roundtrip TCP latencies are as low as 3 microseconds, while InfiniBand latencies can be significantly below 1 microsecond.
Unlike Ethernet, InfiniBand was expressly designed to offload network processing from the CPU as much as possible. Omni-Path Architecture, a proprietary interconnect that may be considered a derivative of InfiniBand, departs from the original InfiniBand philosophy by reinserting the CPU into the network processing pipeline.
The question of exactly where to process the networking protocol has led to distinct approaches of designing and building switches. If the processing is performed wholly or substantially on the network switch or host channel adapter, it is referred to as an “offload” design. By contrast, in an “onload” design, protocol processing is largely performed on the server CPUs. This leads to a trade-off between a more capable network infrastructure (offload) versus incrementally more CPUs on servers (onload).
Even if an onload architecture might be sending a high number of messages per second down the pipes, the tradeoff is that the CPUs are less available to work on the actual user application.
InfiniBand is a mature technology with market support from a broad range of system and component vendors and promoted by the IBTA (InfiniBand Trade Association). Mellanox Technologies is the main provider of networking silicon chips supporting InfiniBand (although Oracle is creating its own switches and adapters), and also sells adapters, gateways and highly scalable switches for the technology. Almost half of the Top500 systems currently use InfiniBand , including the world’s fastest system, the Sunway TaihuLight in China .
Intel’s Omni-Path Architecture is the new kid on the block, with some pedigrees by inheritance of certain technologies from the Cray Aries interconnect and QLogic InfiniBand acquisition, both of which were acquired by Intel in 2012. Intel has moved away from the InfiniBand standards and philosophy by returning network protocol processing functions to the CPU. Currently there are only a handful of customers, so the jury is still out.
The real question for customers is how do the major applications perform on a given interconnect configuration, including switches and topology. Are the apps interconnect-bound? How do they scale as node count increases? What percent of the CPU cores will be doing real application work rather than simply network processing?
What matters most is the achievable application performance and system throughput for a given total cost of ownership (CapEx and OpEx), plus considerations of ease of upgradeability, ease of operation, and reliability.
Assessment And Scoring
The Gartner Magic Quadrant and the Forrester Wave are popular visual representations to help customers understand vendor positions in particular markets. We are introducing a new graphic representation in conjunction with the OrionX Constellation Decision Tool to help people assess different technologies. This goes beyond quadrant and wave methods and evaluates six different parameters that span the three categories of customer, product, and market.
The results of the study and scoring for the HPI market are presented in the graph below:
OrionX scores Mellanox InfiniBand the highest, in recognition of its position as the premier InfiniBand vendor, the maturity of their technology, and a significant installed base.
Next is Intel Omni-Path, whose product portfolio in the HPI market scores well based on a few high-profile early adopters. However, Intel is viewed as simply too new to warrant the higher scores that could come with a long-term presence in the market.
Ethernet is the lowest ranked, though it is used even in many large clusters whose workloads don’t require the very highest performance interconnects. There is obviously differentiation across Ethernet vendors which warrants further analysis.
Stephen Perrenod specializes in corporate strategy for market expansion, sales-marketing alignment, and high performance computing (HPC), cloud computing and big data. Perrenod was director of the HPC business for Asia-Pacific at Sun Microsystems and Oracle, and previously, he was vice president of worldwide sales and marketing for eXludus Technologies, amulticore optimization start-up. He has also held various sales and marketing roles at Cray and SGI and worked in the petroleum industry and in government research labs. He earned a bachelors of science degree from the Massachusetts Institute of Technology (MIT) and a doctorate from Harvard University.
OrionX Constellation reports on the HPI market are available at OrionX.net/research.