NASA@SC19

Supercomputing

What Could Possibly Go Wrong? — The Myth of the Smooth Install

Overview

The Discover supercomputer cluster is the heart of the high-performance computing (HPC) services offered by the NASA Center for Climate Simulation (NCCS). The NCCS provides HPC and data storage services for NASA's Science Mission Directorate. Discover users include hundreds of NASA and university researchers who have come to rely on the regular technology upgrades of the cluster to maximize and accelerate their production of scientific results over the long term.

Project Details

The NCCS has steadily increased the capacity of the Discover cluster since it was first installed in 2006 through the addition of Scalable Compute Units (SCUs). The latest expansion, SCU15, was integrated into Discover in the summer of 2019. This 2-petaflop upgrade increased the cluster’s size to over 4,100 compute nodes containing over 129,000 CPUs and brought its total computing capacity to 6.7 petaflops. That translates into the ability to process over 6 quadrillion operations per second.

SCU15 is the first expansion to the portion of Discover using an Omni-Path high-speed network, while the rest of the cluster operates over multiple InfiniBand high-speed networks. This expansion also increased the number of Intel Skylake CPUs within Discover to over 46,000, allowing users to run higher-resolution models on the system.

Results and Impact

SCU15 represents the 18th upgrade to the Discover supercomputer since its inception. Expanding from a 3.3-teraflop system with a mere five racks of hardware to a 6.7-petaflop system with over 80 racks of hardware has led to growing pains along the way. Everything from physical space to power, cooling, processors, interconnects, and cables provided challenges that were overcome for every successful deployment.

Each SCU installation had its own unique challenges. The NCCS has learned many lessons over the years that were successfully applied to subsequent installations. The installation of SCU15 was no exception. Installed in July 2019, it consists of many components that were similar or identical to the existing hardware deployed throughout the cluster. While the SCU15 vendor was a new partner for the NCCS, the challenge remained the same: to integrate the latest technology upgrade into the cluster as quickly and seamlessly as possible in order to provide additional computing capacity to the NCCS user community as fast as possible. Relying on 13 years of experience performing system upgrades, what could possibly go wrong?

Why HPC Matters

With the latest upgrade to the Discover supercomputer, the NCCS continues to provide state-of-the art HPC resources to the NASA High-End Computing (HEC) Program users in the form of more of the latest, most efficient processors with one of the fastest high-speed interconnects available today. This upgrade, as with all previous upgrades, will enable advances in NASA science and engineering research that would not be possible without computational resources like Discover.

More Information

Bruce Pfaff, NASA Goddard Space Flight Center
bruce.pfaff@nasa.gov