SGI® NUMAlink™ Interconnect Fabric

With microprocessors capable of over one billion calculations per second—less than one nanosecond per operation—fast memory access time is critical in achieving balanced, sustained performance on technical workloads. Data crosses over an SGI NUMAlink switch, round-trip, in as little as 50 nanoseconds—less time than it takes a beam of light to travel 50 feet—compared to 10,000 nanoseconds or more with many commodity clustering interconnects. Furthermore, SGI NUMAlink technology is the only interconnect that provides global shared memory between cluster nodes.

The industry-leading performance of NUMAlink interconnect technology is clear when comparing bandwidth and latency characteristics to other interconnects (Table 1). This translates into better system performance in MPI applications as well as industry standard system benchmarks, such as Linpack (Table 2).

Table 1 - Latency and Bandwidth performance of common interconnect technologies

Technology Vendor MPI latency
usec, short msg
Bandwidth per link
(unidirectional, MB/s)
NUMAlink 4 (Altix) SGI 1 3200
RapidArray (XD1) Cray 1.8 20001
QsNet II Quadrics 2 9002
Infiniband Voltaire 3.5 8303
High Performance Switch IBM 5 10004
Myrinet XP2 Myricom 5.7 4955
SP Switch 2 IBM 18 5006
Ethernet Various 30 100

Bandwidth per link citations are from the following sources:

  1. http://www.cray.com/products/xd1/index.html#RapidArrayInterconnect
  2. http://doc.quadrics.com/Quadrics/QuadricsHome.nsf/DisplayPages/ 81DD13F71CFD762580256EAD0010AA75/$File/Performance.pdf
  3. http://nowlab.cis.ohio-state.edu/projects/mpi-iba/
  4. http://publib-b.boulder.ibm.com/Redbooks.nsf/f338d71ccde39f08852568dd006f956d/55258945787 \ efc2e85256db00051980a?OpenDocument
  5. http://www.myricom.com/myrinet/performance/
  6. http://www-1.ibm.com/servers/eserver/pseries/hardware/whitepapers/sp_switch_perf.pdf

Table 2 - Comparison of Linpack system efficiencies in the November 2004 Top 500 list

System/interconnect Ave. Linpack efficiency for 256P system, %* Sample size, number of systems on list*
SGI Altix/NUMAlink 84 14
HP Superdome 79 18
Various/Quadrics 75 4
Various/Infiniband 75 3 (one system @288P)
Various/Myrinet 63 19
Various/Gigabit Ethernet 59 14

* Linpack Rmax/Rpeak for 256P systems listed on November, 2004 Top 500 list - see www.top500.org