Powering Synthetic Data Generation with HPC and Dask

Overview

Environmental issues concerning water quality and climate change have increased interest in the study of phytoplankton—microscopic marine algae. These organisms make up the foundation of the aquatic food web across the globe and serve several critical roles in recycling carbon and nitrogen. Their responsiveness to nutrient availability and water temperature by changing their distribution and color make them key indicators of change within their ecosystem, and these changes can be observed from Earth-observing satellites.

In order to better understand the observational data, the Biospheric Science team at NASA’s Ames Research Center is developing the Spectral Water Inversion Processor and Emulator (SWIPE) tool, which uses synthetic data to help in the modeling and analysis of the optical properties of water containing algae. However, the process of generating synthetic data is time consuming, so the Biospheric Science team reached out to the Data Science team at the NASA Advanced Supercomputing (NAS) facility at Ames Research Center to help speed up the process.

Project Details

There are four interrelated components to the SWIPE tool: modeling, image processing, big data, and deep learning. The modeling component includes the Equivalent Algal Population (EAP) code that uses a two-layer Mie scattering method (which relates the diameters and optical properties of microscopic particles to the wavelengths of scattered light) to calculate the optical properties of different plankton and phytoplankton species. This synthetic dataset is then used by the image processing component to determine the amount of algae particles in the imaged waterway. EAP data is stored in SWIPE’s big data library, which is used by the deep learning component to generate analytical and predictive models of the water quality.

Generating SWIPE’s synthetic datasets is a time-intensive process, taking over two hours to complete on a laptop computer for just one species of phytoplankton. At the request of the Ames Biospheric Science Team, the NAS Data Science team took a close look at the EAP component with the goal of reducing this bottleneck in their production pipeline. The Data Science team embedded Dask—a flexible open-source Python library for parallel computing—into the SWIPE tool to allow the distribution of the EAP code across multiple processors and nodes on agency high-performance computing systems, like the Pleiades supercomputer at the NAS facility.

Results and Impact

By using the Dask framework, the NAS Data Science team was able to make use of modules such as “dask.delayed” to run SWIPE’s EAP code in parallel, increasing the number of processors allocated to the job. This method reduced the dataset generation time to approximately 18 minutes using Pleiades, compared to two hours on a laptop. Since each Pleiades Broadwell node has 28 processors, approximately 160 datasets can be generated in the same time as a laptop could generate one. With over 70 phytoplankton species being modeled—and more candidate species to be identified—this speed-up significantly reduced the Biospheric Science team’s time-to-solution.

Why HPC Matters

The NAS Data Science team’s efforts to integrate frameworks such as Dask into the SWIPE tool supports the Biospheric Science team’s ultimate goal of building a neural network model that will use observational data to determine water quality from satellite and in-situ imagery. By leveraging the computational power of the Pleiades supercomputer and code parallelization, the two teams were able to increase the rate of dataset generation sixfold, which would not have been possible without access to agency high-performance computing resources and knowledgeable support staff.

What's Next

With the dramatic decrease in the amount of time needed to generate synthetic data, the Ames Biospheric Science team is examining more phytoplankton species as potential candidates to be added to the SWIPE database. Adding the characteristics of additional species will provide researchers with a more accurate determination of water quality for a larger range of waterways around the world.

More Information

• Potential for High Fidelity Global Mapping of Common Inland Water Quality Products
• The Fundamental Contribution of Phytoplankton Spectral Scattering to Ocean Colour
• NAS User Success: Simulating Algae to Model Water Quality in Minutes

Researchers

• Thaddeus Norman, NASA Ames Research Center
  Please enable Javascript to see email address.
• Jeremy Kravitz, NASA Ames Research Center
  Please enable Javascript to see email address.
• Shubha Ranjan, NASA Ames Research Center
  Please enable Javascript to see email address.