High-Performance Computing in the Azure Cloud

Overview

Cloud-based computing provides the possibility of highly dynamic resources that can be spun up on demand to meet computational needs. The NASA Center for Climate Simulation (NCCS) at Goddard Space Flight Center (GSFC) is testing cloud resources as a method for providing computational resources to meet a wide variety of science use cases. One example includes the creation of a digital twin of the Goddard Earth Observing System (GEOS) model, which provides research forecasts for NASA applications. In the event of on-premises system downtimes, the cloud-computing digital twin can provide continuous operations for these types of applications. In addition, many science applications and data analytics require a collaborative environment for NASA and non-NASA contributors. Cloud computing resources could readily be provisioned for these types of requirements.

Project Details

Microsoft has recently deployed Azure offerings targeted directly at high-performance computing (HPC) resources, with top-of-the-line CPUs and InfiniBand-based, remote direct memory access (RDMA) networking—critical for multi-node HPC workloads. These resources provide the NCCS with an ideal prototype for testing both operational HPC capabilities and application performance. This project prototyped the dynamic creation and rapid deployment of HPC clusters in Azure. These clusters were then used to benchmark highly relevant applications to the NCCS and create a cost comparison between on-site resources and the Azure cloud. As an added benefit, the Azure cloud has multiple processor types that can be used for vendor comparison.

Results and Impact

We showed that, while cloud computing can provide commensurate performance for HPC applications, there are many challenges when employing cloud resources for HPC applications. Many basic assumptions of typical data center technologies clash with typical cloud architectures and best practices. We evaluated multiple best practices for rapid deployment of dynamic HPC clusters in the cloud and developed methodologies for quickly creating dynamic HPC clusters.

Why HPC Matters

For static workloads, cloud-based resources are not yet competitively priced versus on-premises compute clusters. However, the rapid ability to stand up or expand resources provides a solution for production workloads and meets a wide variety of requirements, including on-premises downtimes. Alternately, cloud-based resources can provide workspaces that are isolated from strict security environments, allowing a location for collaboration between government and non-government personnel for low-sensitive, scientific data exploration. The ability to leverage these capabilities for an operational center similar to the NCCS provides users with more options to accelerate their scientific research.

What's Next

Recent offerings from Amazon that include Elastic Fabric for higher-performance networking and managed Lustre environments offer interesting capabilities. In addition, the deployment frameworks recently developed on Azure will be evaluated for their cross-platform capabilities and directly compared against the features and performance of alternate cloud vendors.

More Information

• See the poster for this demonstration (PDF, 686KB). As shown in the NASA booth at SC19.
• Download the demonstration slide deck (PPSX, 10.5MB) as presented in the NASA booth at SC19. Click "Read Only" after launching presentation.