Navigating a Sea of Climate Data

Overview

NASA is developing new big data analytic capabilities to improve the productivity of weather/climate research. Investigators seeking to understand Earth science phenomena face several daunting challenges when trying to utilize NASA's extensive collections of observational and computer model data. The initial steps of such research are to first identify the occurrences of a phenomenon of interest, such as hurricanes or heat waves, and then to extract the corresponding subsets from the available data. Unfortunately, the existing two-staged practice of archiving data into files before distribution only permits searching on the metadata of the files (for example, the time and geolocation ranges of the files) rather than directly on the data the files contain (temperature, pressure, wind, and so on). Researchers are therefore forced to download voluminous amounts of data to perform expensive search and analysis using local computing facilities. The new technology addresses these inefficiencies by not only reducing the amount of data transferred, but also by facilitating parallel server-side data analysis.

Project Details

The Automated Event Service (AES) is an integrated system of software tools designed to meet the "Four V" challenges of Big Data—enormous volume, copious variety, demand for velocity, and ensured veracity—by providing powerful search and analysis capabilities. AES catalogs weather and climate data into a next-generation parallel database, SciDB, which is specifically designed to support multidimensional scientific array data. AES is being developed at NASA's Goddard Space Flight Center to demonstrate the value of employing big data technology for weather and climate research. Scientists can use a familiar, lightweight Python interface to leverage SciDB's array query language and array functional language in order to efficiently conduct a diverse variety of search and analysis. Custom user defined operators can be implemented to extend SciDB's capabilities as needed.

Results and Impact

The power of AES was readily demonstrated in our Blizzard use case. A search in the Modern-Era Retrospective Analysis for Research and Applications (MERRA) project data for winter 2010 blizzards over the continental United States took only seconds to complete. The results were verified by checking the National Weather Service reports and incorporating NEXRAD radar estimates of precipitation for visual inspection. An expansion of the search to 30 years of global data finished in hours, when previous approaches required more time just to download the necessary data.

Why HPC Matters

To support a wide array of scientific queries, AES must host large volumes of data from a wide variety of data collections stored on disk. Fast response time requires a powerful system with high bandwidth. For the prototype system, we are utilizing the High-Performance Data Analysis Platform at the NASA Center for Climate Simulations, a 30-node system with 32-gigabyte RAM memory, extremely high bandwidth, and 1.2 petabytes of storage.

What's Next

Plans are underway to increase the power of AES through increased storage and computing resources. Project researchers are also exploring the feasibility of using cloud-based resources to increase capabilities by 1 - 2 orders of magnitude. While the intermittent demand that characterizes typical use scenarios is highly favorable to cloud computing, the requirement for large, persistent data storage can lead to unacceptable cloud storage costs. A production AES system would likely require well over one petabyte of storage.