Skip to main content
NASA Advanced Supercomputing (NAS) Division World-class modeling, simulation, and supercomputing in support of the NASA mission

Intelligent and Progressive Scientific Data Management

AMS Seminar Series

Speaker: Qing Liu, New Jersey Institute of Technology / X-Byte Research

Presentation

Seminar Slide Deck (PDF-2.7MB)

Abstract

As simulation-based scientific discovery advances to exascale, a major question that the community is striving to answer is how to co-design data storage and complex physics-rich analytics in a way that the time to knowledge can be minimized for post-processing. A particular challenge is how to accommodate a broad spectrum of data analytics needs–particularly those that become clear only until very late during the post-processing, a scenario where existing methods, such as in situ processing, are unable or less effective in supporting data analytics. As HPC storage systems have become deeper and more complex with the recent addition of NVMe, die-stacked memory, and burst buffer, it requires fundamentally rethinking new paradigms and methods for data storage and analysis.

This talk aims to address the issue of I/O bottleneck for data analytics over parallel storage systems, which are shared by multiple applications in a non-exclusive manner. At the core of our work called MIO is a coordinated cross-layer approach that reacts to storage interference from both storage and application layers. By decomposing and distributing analysis data across the storage hierarchy, data analytics can adapt to the interference by reducing or completely avoiding access to lower tiers whenever there is a high interference, while maintaining a prescribed error bound to limit the information loss. Meanwhile, proper actions are also taken at the storage layer to ensure sufficient bandwidth is allocated for retrieving an augmentation, which is based upon the cardinality and accuracy of the augmentation as well as the nature of an application. We evaluated three real-world data analytics, XGC, GenASiS, and CFD, on Chameleon, and quantitatively demonstrate that the I/O performance can be vastly improved, e.g., by 52% versus no adaptivity and 36% versus single-layer adaptivity, while maintaining acceptable outcomes of data analysis.

Biography

Photo of Qing Liu
Qing Liu is an Associate Professor in the Electrical and Computer Engineering department at the New Jersey Institute of Technology and holds a Joint Faculty Appointment with Oak Ridge National Laboratory. He is the CTO at X-Byte Research, a technology firm that builds scalable system software for HPC applications. Prior to that, he was a Staff Scientist with the Computer Science and Mathematics Division at Oak Ridge National Laboratory. His research interests include high-performance computing and scientific data management. Liu received NSF CAREER award in 2022 and R&D 100 award as a principal investigator in 2013 for his contributions to adaptable I/O systems for Big Data applications. His research has been highlighted by DOE, DOD, Oak Ridge Leadership Computing Facility, HPCWire, and has won the best paper finalist for IPDPS’2018 and a best paper award for NAS’2019.