High-End Computing at NAS: Furthering NASA Science and Engineering Goals

Overview

The High-End Computing Capability (HECC) Project, operated by the NASA Advanced Supercomputing (NAS) Division at Ames Research Center, provides an agency-wide compute environment focused on enabling science and engineering discoveries across all mission areas. As the agency increasingly leverages high-fidelity modeling and simulation and large-scale data analysis to pursue its challenging missions, NASA's supercomputing requirements will continue to grow over time. To meet these needs, HECC regularly and significantly upgrades the supercomputing resources it hosts for the agency, and provides a comprehensive set of services, including code optimization, data analysis and visualization, and customizable storage and networking solutions.

Project Details

Over the past year, NAS teams have twice expanded NASA’s flagship Pleiades supercomputer. The first expansion added 2,088 nodes of Intel E5-2680v2 (Ivy Bridge) processors, bringing the total number of Ivy Bridge nodes to 5,400. In the second expansion, 1,080 nodes of Intel’s newest-generation E5-2680v3 (Haswell) processors were incorporated into the supercomputer. Together, these expansions have increased the overall peak performance of Pleiades by more than 55% since 2013—from 2.9 to 4.5 petaflops.

The NAS facility’s hyperwall visualization system was also upgraded with 128 new Ivy Bridge compute nodes, each containing an NVIDIA GeForce GTX 780 Ti graphics card, to drive the system’s 128 screens. Each node can display, process, and share data. This feature enables researchers to see detailed results from many related simulations at once—such as images of a proposed launch vehicle design at multiple speeds and flight attitudes—or view a single high-resolution image across all of the hyperwall’s screens.

Other hardware enhancements made this year include increasing the Lustre storage capacity to 20 petabytes (PB), and upgrading the 115-PB-capacity Spectra Logic archive tape libraries with new robotics to speed load times and provide a path for future growth.

Recently, NAS systems and visualization teams worked with researchers from the consortium for Estimating the Circulation and Climate of the Ocean (ECCO) to run a global ocean simulation at an unprecedented resolution of 1/48-degree (48 grid points per degree of latitude and longitude) across 70,000 Pleiades cores. To enable the run, the teams resolved several highly complex hardware and software issues including removing I/O bottlenecks, optimizing SGI’s Message Passing Toolkit libraries, and fixing InfiniBand network problems that had also affected many other high-end computing centers.

Results and Impact

At 3 petabytes, the output of the ECCO simulation is the largest individual dataset ever produced at NAS. The simulation’s resolution is so high that it can only be fully displayed using the 128-screen hyperwall at the system’s full resolution of 246 million pixels. To help researchers analyze this enormous amount of data, NAS visualization experts developed a utility that enables them to navigate through multiple layers of the animation in order to examine various ocean depths and scalar values. The work achieved by the NAS teams will help provide researchers with significant new insight into ocean circulation, temperature, salinity, and sea level, and the way in which these phenomena interact on a global scale.

The NAS division supports hundreds of researchers and engineers performing large-scale modeling, simulation, and analysis to help solve some of the world’s most complex problems. By providing a well-balanced combination of state-of-the-art computing platforms and integrated services, NAS will continue to enable significant, timely impacts across NASA's missions in Earth and space science, aeronautics research, and human and robotic space exploration.