Extending and Exploring GEOS Performance and Portability

Overview

The Goddard Earth Observing System (GEOS) is a modular group of models developed by NASA’s Global Modeling and Assimilation Office (GMAO) for research in near-real-time forecasting, seasonal-to-decadal analysis and prediction, reanalysis, and high-resolution mesoscale modeling. As GEOS runs at higher and higher resolutions, understanding how its computational environment—operating systems, compilers, processors, network fabrics—affects its performance and portability becomes crucial. Researchers are also exploring ways to increase performance by using alternative programming paradigms such as vectorization, advanced MPI, and OpenMP to improve scaling when moving to finer scales.

Project Details

GEOS as a model can run a single atmospheric sounding up to hundreds of millions of grid cells on hardware as small as a laptop to tens of thousands of cores on NASA supercomputers. To determine the effects on performance and portability, we are testing GEOS in a range of software environments—from macOS to Linux, from the freely available GNU compilers and Open MPI to high-performance stacks from Intel and Hewlett Packard Enterprise (HPE) on different network interconnects.

We are also exploring various optimizations to enhance performance. Work has included testing the effects of using vectorization, hybrid MPI+OpenMP, and accelerators. In addition, we are testing asynchronous client-server strategies to help speed up I/O reads and writes as the model runs. Finally, we are integrating modern software engineering paradigms like Git, CMake, and continuous integration to allow us to better maintain and develop the code base in a systematic manner.

Results and Impact

Testing shows that the choice of compiler and MPI stack can have a major effect on GEOS performance. As predicted, vendor-provided MPI stacks are generally the best on their own systems—HPE MPI on HPE InfiniBand clusters and Intel MPI on Omni-Path clusters, while Open MPI allows portability.

We have found that using single-precision math with vectorization enables the model dynamics to run faster with no significant effect on model results. Our use of hybrid OpenMP+MPI for the dynamics and moist convection physics has shown promising results as well. Taking these steps should help overcome the current scaling limits found in a pure MPI environment when running the model at high resolution and allow us to meet production needs.

Why HPC Matters

High-performance computing (HPC) at NASA is essential for both research and realistic forecasting of weather and climate. Without large computational clusters, we would never be able to run super-high-resolution simulations demanding thousands of processors, terabytes of memory, and petabytes of disk space. By exploring GEOS’s behavior on various hardware and software, we can more efficiently use supercomputing resources to meet performance goals.

What's Next

Future work on GEOS will continue extending the use of OpenMP to more effectively scale at high core count. We also know that I/O is an ongoing concern and will give attention to strategies such as prefetching input data as well as exploring different compression schemes to help lower disk space use as output data grows with model resolution.

We are also continuing to introduce modern software engineering practices to GEOS. The ability of GEOS to run portably at all scales will enable the use of emerging technology like HPC containers and continuous integration, which in turn will allow us to ensure that code changes both comply with the Fortran Standard and do not have undesired effects on model output.

More Information

• Global Modeling and Assimilation Office
• See the poster for this demonstration (PDF, 1.9MB). As shown in the NASA booth at SC18.