Building Petabyte Data Production Systems with the NASA Earth Exchange

Overview

The NASA Earth Exchange (NEX) is a platform for scientific collaboration and knowledge sharing among the Earth science community. NEX combines state-of-the-art supercomputing, Earth system modeling, workflow and knowledge management, and NASA remote sensing data feeds to provide a complete work environment in which users can explore and analyze large datasets, run modeling codes, collaborate on new or existing projects, and quickly share results with other scientists.

To support a growing number of projects that are developing large-scale Earth science data products, the NEX team has developed a data production system that that links high-performance computing (HPC) with a cloud environment. The system represents both the production rules and the execution of these rules as a directed graph, from which users can analyze and reproduce scientific results.

Project Details

In order to support large-scale data production, the NEX platform has been extended to simplify the definition and deployment of production workflows; enable performance analysis of individual workflow components; and improve quality monitoring and scientific reproducibility. These capabilities enable research teams to build complex non-linear production pipelines, and are already in use for several projects—ranging from the application of machine learning for counting trees across North America, to developing consistent monthly and annual global higher-level satellite data products at 30-meter spatial resolution—the first of their kind at this resolution.

Through semantic integration with the NEX knowledge and data management components, the NEX production system can answer additional queries about data quality, resource usage, or related publications. The output of the data processing pipeline is a data product and a graph that represents the entire provenance of each individual data granule.

Results and Impact

The NEX system makes it possible for small science teams to execute projects that would not have been possible previously. Examples include:

• Production pipeline for Landsat and NASA Global Imagery and Browse Service – 37 steps, 7 petabytes of data, and 670,000 processor hours.
• NEX Downscaled Climate Projections (NEX-DCP30) and NEX Global Daily Downscaled Projections (NEX-GDDP) pipeline – 40 terabytes (TB) of data and 320,000 processor-hours.
• Estimating biomass by counting trees across the United States by using machine learning and computer vision – 40 TB of data/100,000 compute hours.

Why HPC Matters

The Pleiades supercomputer, combined with the massive data storage and high-speed network resources at the NASA Advanced Supercomputing (NAS) facility, enables the NEX team to engage large scientific communities and provide them with large-scale modeling and analysis capabilities not previously available to most scientists. Over the last year, NEX projects used more than 860,000 processor-hours on Pleiades and processed more than a petabyte of data.

What's Next

We plan to engage users and make it substantially easier to build big data production pipelines in the HPC environment. We will also increase our engagement with machine learning and data mining communities, scale up production of long-term data records for Landsat, create a prototype production system for data from the European Space Agency's Sentinel-2 mission, and extend the data pipeline to NASA's Global Imagery and Browse Service (GIBS).