Taking on Big Data with Climate Analytics-as-a-Service

Overview

To help address the big data challenges of climate science, the Computational and Information Sciences and Technology Office (CISTO) at NASA's Goddard Space Flight Center is developing a Climate Analytics-as-a-Service (CAaaS) capability. CAaaS is a specialization and extension of the Software-as-a-Service (SaaS) concept enabled by cloud computing.

Project Details

NASA's Modern-Era Reanalysis for Research and Applications (MERRA) Analytic Services (MERRA/AS) is defining the critical software components needed to successfully implement the CAaaS capability. MERRA/AS enables Hadoop MapReduce analytics over the data collection by bringing together high-performance computing, scalable data management, and a domain-specific Climate Data Services Application Programming Interface (CDS API).

The agency's High-Performance Data Analysis Platform (HPDAP) is an example of the type of compute-storage platform required to support CAaaS. The HPDAP comprises a high-speed InfiniBand network, high-performance file systems and object storage, and virtual system environments specific for data-intensive science applications.

Results and Impact

MERRA integrates data from satellites and other observational platforms with the Goddard Earth Observing System Model, Version 5 (GEOS-5) to produce a temporally and spatially consistent synthesis of climate variables that are not easily observed. It covers the years 1979 to the present—essentially the entire satellite era. The breadth of MERRA variables, which include atmosphere, ocean, and land surface products, makes MERRA ideal for investigating climate variability and for use in an expanding array of applications.

MERRA/AS dramatically improves customer access to MERRA data. It moves much of the work traditionally done on the client side to the server side, close to both the data and large compute power. This approach reduces the need for large data transfers and provides a platform to support complex server-side data analyses.

Over the past year, MERRA/AS (including the CDS API) has undergone beta testing with government, corporate, and university projects and users. Participants' varying research and application areas include plant sciences, reanalysis studies, terrestrial ecology, agricultural forecasting, wildfire recovery, and invasive species forecasting.

Why HPC Matters

Powerful computing resources are crucial for on-demand analytic processing and querying of massive datasets. MERRA/AS provides access to 26 key weather and climate variables (with more than 400 advanced diagnostic variables) from the original MERRA dataset of 160 terabytes. The latest version, MERRA2, is currently running at the NASA Center for Climate Simulation (NCCS). This update is using the latest version of GEOS-5, which adds land precipitation data to set more realistic initial conditions, and ingests aerosol observations to allow air quality and other studies. Scientists anticipate a MERRA2 dataset of 350 terabytes with more than 1,000 variables.

The HPDAP currently offers 960 computer processor-cores, 2 petabytes of storage, and up to 48 virtual machines for data analysis customers. To harness the HPDAP capabilities, NASA researchers preparing for the Arctic Boreal Vulnerability Experiment (ABoVE) built surface water maps of Alaska and western Canada in a mere 6 weeks—work that would have taken at least 9 months using a workstation cluster. With other applications experiencing similar speed-ups within this reconfigurable environment, the HPDAP has the agility to meet customers' increasing and changing needs.