Market Overview

With the availability of low cost, high-volume servers running open source software and utilizing newer technologies such as clustering and high performance interconnects, affordable high performance computing is now ready to take off in many environments such as academia and research labs for numeric intensive computing and visualization to wall street for financial decisions, health science for drug discoveries and all the way to Hollywood to make surrealistic movies.

According to IMEX Research, applications market may be divided into four major market segments:
1. Online Transaction Processing (OLTP)
2. Decision Support System (DSS)
3. High Performance or Numeric Intensive Computing (HPC)
4. Data Streaming/Digital Living (STR)

Rise of Clusters in HPC

There has been a phenomenal rise in the use of clustering technology to build high performance computers in the last ten years. Clustering technology orchestates computing amongst low latency, low cost compute nodes via high bandwidth interconnects necessary to make high performance computer feasible.

Previously the largest supercomputer boasted a performance of 143 Gflops without the use of clustering technology but costing an astounding $40 millions so was used only by Government National Labs and few large corporations. Today clustering has enabled supercomputers to run upto 71 Tflops less than $4000 (a price/ performance inprovement of 5,000,000x). This has allowed HPC to start penetrating a broad spectrum of markets such as medicine, bio-informatics, aerospace, life sciences, weather forecasting, defense, movies and financial markets.

Market Leaders in HPC

Scientists and researchers have long been in forefront for building low cost high performance computers and have successfully led the way in the adoption of clustering technology. On the commercial side, IBM is the market leader in high performance computing with 44% market share and Hewlett Packard comes second with 34% shares followed by NEC and Sun amongst the top 500 supercomputers. Dell by virtue of low cost direct market model is aggressively targeting the high performance computer market via the rack mount and blade server technology.

HPC Architecture and Challenges

The main goal of HPC architecture is to maximize compute time and reduce or hide latency (messaging time between cluster nodes) while keeping the highest possible cross section throughout. The challenge in HPC architecture and implementation are multi-level because the entire HPC communication stack needs optimization as shown below.

Fabric based Cluster Architecture

The fabric based architecture that interconnects multiple nodes relies on a myriad of pertinent fabric technologies. Prudent choice from amonst the emerging technologies in System Fabric (Quadrics, Myrinet,SCI, Infiniband), Network Fabric (Ethernet, Wi-Fi),Storage Fabric (SCSI,Fiber Channel, ISCSI) and Management Fabric (SNMP,IETFI,CIM,SMI-S,SMASH)pose a continuous challenge to successfully provide low latency, and high bandwidth data transfer.

The World of HPC Applications from Academia to Wall Street

High performance computing embrace is ripe to a broad spectrum of markets and applications:

1. Research and Development
   Research in drug discovery, diagnostics, information-based medicine, and bioinformatics are the mainstays of Health Sciences.
2. Enterprise Optimization
   Energy (Oil and gas exploration/production), Automotive, Aerospace, Health Sciences, Manufacturing(CAD,CAM,CAE), electronics
3. Business Analytics
   Wall Street Financial Risk Management (high volume data flow and transactions), Insurance, Interactive Compliance, and online Management using high performance dash boards.
4. Government
   Scientific research, classified/defense, and weather/environmental sciences.
5. Entertainment
   Digital content creation, Video on Demand