Cray Message Passing Toolkit (CMPT) is a collection of high-performance software libraries within the Cray Programming Environment (CPE) that enables parallel applications to communicate and exchange data efficiently across nodes in a Cray system with Slingshot interconnect, such as Athena which uses Slingshot 11. CMPT includes MPICH, PMI, OpenSHMEMX, DSMML, and MPIxlate. CPE version 25.09 includes MPICH 8.1.33 and 9.0.1, PMI 6.1.16, OpenSHEMEMX 11.8.0, DSMML 0.3.1, and MPIxlate 1.0.9.

Note: MPICH 8.1.33 implements the MPI-3.1 standard. MPICH 9.0.1 includes support for MPI 4.0 standard features. MPICH 8.1.33 is the current default on Athena.

As the name suggests, Cray MPICH is based on MPICH. If your existing executable was built with a dynamically linked MPI library that is Application Binary Interface (ABI)-compatible with the Cray MPICH library, it might be able run on Athena without recompiling and linking with the Cray MPICH library. For example, Intel MPI is also based on MPICH, and is ABI-compatible with the Cray MPICH library. On the other hand, OpenMPI and HPE/SGI MPT library are not ABI-compatible with MPICH.

This article provides brief information about using the CMPT modules cray‑mpich, cray‑mpich-abi and cray‑mpixlate to build or prepare an MPI executable, together with the cray‑pals module from the Cray Parallel Application Launch Service (PALS) to manage job startup and process launch on Athena.

Building Applications with the cray-mpich Module

There are three methods of using cray-mpich to build an application.

Method 1: Explicitly Invoking the Compiler and Linking to the Cray MPICH Library

To explicitly build an executable using the cray-mpich module and a compiler from either PrgEnv‑cray, PrgEnv‑intel, PrgEnv‑aocc or PrgEnv‑gnu:

module load PrgEnv-cray  (or PrgEnv-intel, PrgEnv-aocc, PrgEnv-gnu)
module load cray-mpich   (or cray-mpich/9.0.1)

name_of_compiler_driver <compiler flags> application_source_file.[f90,c,cpp]
-I/path-to-include-directory-of-other-library
-L/path-to-lib-directory-of-other-library -lname_of_other_library
-I${MPICH_DIR}/include
-L${MPICH_DIR}/lib -lmpi   (for C/C++ applications)
or 
-L${MPICH_DIR}/lib -lmpi -lmpifort (for Fortran applications)

The name of the compiler driver varies as follows:

PrgEnv-cray : crayftn, craycc, crayCC
PrgEnv-intel: ifort, icc, icpc, ifx, icx, icpx 
PrgEnv-aocc : flang, clang, clang++
PrgEnv-gnu  : gfortran, gcc, g++

The ${MPICH_DIR} can be replaced with ${CRAY_MPICH_PREFIX} or ${CRAY_MPICH_DIR}, since all three of them are defined in cray-mpich and they point to the same directory.

Depending on which PrgEnv is used {-cray, intel, aocc, gnu} and whether cray-mpich/8.1.33 or cray-mpich/9.0.1 are loaded, the actual path of ${MPICH_DIR} varies. For example, if using PrgEnv‑cray and cray‑mpich/8.1.33, then ${MPICH_DIR} is /opt/cray/pe/mpich/8.1.33/ofi/cray/20.0.

Note: Athena is an HPE Slingshot 11 system that only supports libfabric — a low-level communication library also known as the OpenFabrics Interfaces (OFI). The Unified Communication X (UCX) framework for communication is supported in Slignshot 10 systems, but not Slingshot 11 systems with Cray MPICH.

Method 2: Using the Cray MPICH Wrapper Scripts

Cray MPICH also provides standard MPI wrappers (mpicc, mpicxx, mpic++, mpifort, mpif77, mpif90) for invoking compiler and linking the proper Cray MPICH library in accordance with the compiler that is used (cray, intel, aocc, gnu).

Note: mpic++ is a soft link to mpicxx. mpif77 and mpif90 are soft links to mpifort.

Usage:

% module load PrgEnv-cray (or PrgEnv-intel, PrgEnv-aocc, PrgEnv-gnu)
% module load cray-mpich  (if not already loaded)
% mpifort <FFLAGS> source.f90
% mpicc <CFLAGS> source.c
% mpicxx <CXXFLAGS> source.cpp

If other non-MPI libraries are needed for building the application, they need to be explicitly added to the  mpifort, mpicc, mpicxx command line as shown in Method 1.

When using the MPI wrappers, the variables MPICH_FC, MPICH_CC and MPICH_CXX can be used to change the choice of compilers. For example, with PrgEnv-intel, mpifort, mpicc, and mpicxx default to using ifx, icx and icpx regardless of which of intel, intel‑classic, intel‑oneapi you have loaded. If you want mpifort, mpicc, and mpicxx to use ifort, icc, icpc, do the following before invoking mpifort, mpicc or mpicxx:

With csh:

% setenv MPICH_FC ifort
% setenv MPICH_CC icc
% setenv MPICH_CXX icpc

With bash:

% export MPICH_FC=ifort
% export MPICH_CC=icc
% export MPICH_CXX=icpc

Method 3: Using Cray's Compiler/Link Wrapper Scripts

When a cray-mpich module is loaded together with a compiler provided under either PrgEnv-cray, PrgEnv-intel, PrgEnv-aocc, or PrgEnv-gnu, the Cray wrappers ftn, cc, CC will automatically link the Cray MPICH library (‑lmpich or simply ‑lmpi for C/C++ and ‑lmpi ‑lmpifort for Fortran) and other Cray libraries (for example, Cray Scientific Library) to ensure compatibility and optimized performance.

The actual -lmpi and -lmpifort linked in depend on the compiler used. For example, if a Cray compiler (crayftn, craycc, crayCC) is used, then ‑lmpi_cray and ‑lmpifort_cray are linked in.

Usage:

% module load PrgEnv-cray (or PrgEnv-intel, PrgEnv-aocc, PrgEnv-gnu)
% module load cray-mpich  (if not already loaded)
% ftn <FFLAGS> source.f90
% cc <CFLAGS> source.c
% CC <CXXFLAGS> source.cpp

If your application needs other libraries not included in the Cray Programming Environment, you must explicitly add them to the ftn, cc, CC command line as shown in Method 1.

Here are a few useful Cray MPICH and LIBFABRIC environment variables to consider:

• MPICH_ENV_DISPLAY=1: Causes rank 0 to display all MPICH environment variables and their current settings at MPI initialization time.


• MPICH_VERSION_DISPLAY=1: Causes MPICH to display the HPE Cray MPI version number as well as build date information; helps to track issues with different MPI versions.


• MPICH_RANK_REORDER_DISPLAY=1: Causes rank 0 to display which node each MPI rank resides in. The rank order can be manipulated via the MPICH_RANK_REORDER_METHOD environment variable.


• MPICH_MEMORY_REPORT=n:
  • n=1: prints a summary of the min/max high water mark and associated rank to stderr.
  • n=2: outputs each rank's high water mark to a file as specified using MPICH_MEMORY_REPORT_FILE.
  • n=3: does both 1 and 2.


• MPICH_OFI_VERBOSE=1: Helps for debugging by displaying more verbose output during MPI_init to verify which libfabric provider has been selected, along with the name and address of the NIC being used.


• MPICH_OFI_USE_PROVIDER: On Slighshot 11 systems, the default is the cxi provider. Other libfabric providers (such as "sockets") can be requested for debugging purposes.

  Note:  cxi stands for Cassini fabric provider in libfabric.

• FI_PROVIDER: Controls what providers to load.


• FI_LOG_LEVEL=warn & FI_LOG_PROV=cxi: Causes the Cassini provider to log messages which help to debug issues.


• FI_CXI_RX_MATCH_MODE: Specifies the receive message matching mode to be utilized; possible choices are hardware, software, hybrid.

  Set this variable to hybrid or software if your application fails to run and the error message includes a recommendation to set this variable.



• MPICH_OFI_NIC_MAPPING: Specifies the precise rank-to-NIC mapping to use on each node. This is evaluated only if the MPICH_OFI_NIC_POLICY variable is set to USER.


• MPICH_OFI_NIC_POLICY: Selects the rank-to-NIC assignment policy used by Cray MPI. Each MPI rank will be assigned to exactly one NIC. There are five available options: [BLOCK | ROUND-ROBIN | NUMA | GPU | USER].


• MPICH_OFI_NIC_VERBOSE: If set to 1, verbose information pertaining to NIC selection is printed at the start of the job.


• MPICH_OFI_NUM_NICS: Specifies the number of NICs the job can use on a per-node basis.

For more information, read man intro_mpi and the libfabric man fabric and man fi_cxi.

Running Applications with the cray-pals Module

Loading a cray-mpich module provides access to scripts such as mpifort, mpif77, mpif90, mpicc, mpicxx, mpic++ as well as various libmpi* for building MPI applications. However, cray-mpich does not provide mpiexec for use during runtime. A cray-pals module is needed during runtime to access the mpiexec executable in the parallel application launch service (PALS).

Note: Make sure cray-pals is loaded before launching your application with mpiexec.

Usage:

% module load cray-pals    (if not already loaded)
% mpiexec [options] your_executable

Here are some useful mpiexec options to consider:

• -n, -np, --np: Number of processes to start. Default is 1.


• -ppn, --ppn: Processes/ranks per compute node.


• -hostfile, --hostfile: File containing list of host names. Default for <hostfile.txt> is $PBS_NODEFILE. For example:

  mpiexec [-hostfile|--hostfile] <hostfile.txt> <application>



• --cpu-bind [verbose],[core,depth,none,numa,socket,core,thread,list,mask]: Bind ranks to CPUs. Default is thread. For example:

  mpiexec --cpu-bind core -n 8 <application>



• --mem-bind [none, local, list, mask]: Bind ranks to NUMA nodes.


• -env or --env: Set an environment variable. For example:

  mpiexec [-env|--env]  <ENV_VAR_NAME>="ENV_VAR_VALUE" <aplication>



• -l, --label: Add node and rank labels output.


• --line-buffer or --no-line-buffer: Line buffered mode for stdio, to avoid interleaving lines from multiple ranks. Default is ‑‑no‑line‑buffer to allow fully buffered stdio.


• --restart: mpiexec to restart application if it detects that an application aborted. Default value for ‑‑restart is 16.


• -v, --verbose: More verbose output.

Running Applications Built with Intel MPI Without Rebuilding the Executable

An existing executable built with the Intel MPI library might be able to run on Athena without having to rebuild it with the Cray MPICH library. To try this, use the cray-mpich-abi module. For example:

1. Load all modules and/or set any variables required to run the application, except the Intel MPI module.
2. Load the Cray MPICH module cray-mpich-abi instead of cray-mpich.
3. Load the cray-pals module.
4. Launch the application.

Note: If the existing executable was built with an Intel MPI version older than version 5.0 (released around 2014), use cray-mpich-abi-pre-intel-5.0 instead of cray-mpich-abi.

For example:

% module switch PrgEnv-cray PrgEnv-intel   
   (if you executable needs some libraries from Intel compiler)

% setenv LD_LIBRARY_PATH \
    "/nasa/intel/oneapi/mkl/2023.2.0/lib/intel64:$LD_LIBRARY_PATH" 
    (if MKL is needed. Note that the command line is too long to be formatted 
as one line, so it is broken with a backslash [\].)
% module switch cray-mpich cray-mpich-abi
% module load cray-pals
% mpiexec [options] executable-built-with-intel-mpi

Translating Executables Built with OpenMPI or HPE/SGI MPT to Cray MPICH

MPIxlate, which stands for MPI Translate, enables C and C++ applications compiled using an MPI library that is not binary compatible with Cray MPICH to be run without recompilation on supported HPE Cray systems.

MPIxlate transparently translates OpenMPI ABI and HPE/SGI MPT ABI to MPICH ABI at runtime.

Usage:

1. Load all modules and/or set any variables required to run the application, except the OpenMPI or HPE/SGI MPT module.
2. Load the Cray MPICH module cray-mpich.
3. Load the MPIxlate module cray-mpixlate.
4. Load the cray-pals module.
5. Launch the application using MPIxlate as a pass-through intermediary.

For example, to translate an application built with OpenMPI with an ABI major number 40 (libmpi.so.40) to Cray MPICH with an ABI major number 12 (libmpi_cray.so.12), do:

% mpiexec [mpiexec options] mpixlate -s ompi.40 -t cmpich.12 a.out ...

To translate an application built with HPE/SGI MPT 2.25 to Cray MPICH with an ABI major number 12, do:

% mpiexec [mpiexec options] mpixlate -s sgi.225 -t cmpich.12 a.out ...

Note: With MPIxlate 1.0.9, only ompi.40 and sgi.225 are supported. If your C/C++ executable was built with HPE/SGI MPT 2.30, try -s sgi.225.

For more information about MPIxlate and its limitations, see man mpixlate.