Turbulence in fluid dynamics is a subject of great complexity and practical importance, and the source of an insatiable demand for extreme scale computing power in pursuit of fundamental advances in understanding and modeling. Direct numerical simulations (DNS) of turbulence using Fourier pseudo-spectral methods have been a vital research tool and serves as a benchmark for advances in leadership-class computing dominated by heterogeneous GPU platforms, which often provide both opportunity and challenge. In this talk, we discuss how different machine-specific programming models have allowed us to extract maximum benefit from two successive top-ranked GPU platforms, namely Summit and Frontier, at the Oak Ridge Leadership Computing Facility (OLCF).
We first review the development and application of a batched-asynchronism algorithm on Summit (Ravikumar, Appelhans and Yeung, SC19 Conference), where overlapping between GPU operations, host-device data movement and MPI communication on different batches of the data is an effective response to machine’s property that the CPU has a memory several times larger than GPU. We then focus on Frontier, which is currently the world’s fastest and first Exascale machine. The two most important properties of Frontera are that (I) the CPU and GPU have the same amount of memory, and that (II) the network cards are built directly into the GPU hardware. These properties allow us to perform almost all computation on the GPU, with only minimal host-device data movement, and to perform communication directly and efficiently on the GPU, as well (Yeung, Ravikumar, Nichols and Uma-Vaideswaran. Under review at Comput. Phys. Commun., 2024). Our work on Summit and Frontier has enabled record-breaking problem sizes, at 184323 (6 trillion) and 327683 (35 trillion) grid points, respectively. The version of pseudo-spectral algorithm is, at its very core, still one based on Rogallo 1981 (NASA Tech Memo. 81315 at Ames).
Biography
P.K. Yeung received his bachelor’s and master’s degrees from the University of Hong Kong, and a Ph.D. in Mechanical Engineering from Cornell University in 1989. He has been on the faculty in aerospace engineering at Georgia Tech since 1992. Professor Yeung’s research interests are focused on incompressible turbulence, including similarity scaling, extreme events, Lagrangian intermittency, dispersion, mixing, as well as electromagnetic forces; and the pursuit of fundamental understanding via direct numerical simulations at the highest resolutions feasible at a given time. He has been active in the computational science community, and has received large resource allocations at many national supercomputer centers, including currently those at the Oak Ridge National Laboratory and at the University of Texas at Austin. Professor Yeung’s research is primarily funded by the U.S National Science Foundation, including fluid dynamics and cyberinfrastructure programs. He is a Fellow of the American Physical Society and of the American Society of Mechanical Engineers.
Kiran Ravikumar obtained his master’s and doctoral degrees in aerospace engineering from Georgia Institute of Technology in 2021. He is a research scientist for the Science and Technology Corporation working in the Computational Aerosciences Branch of the NASA Advanced Supercomputing Division at NASA Ames Research Center. He is one of the developers of Launch Ascent and Vehicle Aerodynamics (LAVA) which aims at solving challenging aerodynamics, aeroacoustics and aeroelastic problems with high-fidelity numerical tools. His PhD dissertation was on “Extreme-scale computing and studies of intermittency, mixing of passive scalars and stratified flows in turbulence”. His Ph.D. research involved developing a pseudo-spectral direct numerical simulation code to study turbulence on leadership class supercomputers like Summit and Frontier. After graduating he worked as a Software Engineer at Hewlett Packard Enterprise where he developed numerical libraries like BLAS and LAPACK for AMD and NVIDIA GPU targets. He is passionate about high-performance computing, GPUs, and leveraging optimized numerical algorithms for computational fluid dynamics.
