Running Apptainer in hybrid MPI mode allows you to call mpiexec or a similar launcher on the apptainer command itself when you execute an Apptainer container with an MPI code. The MPI process outside of the container will then work in tandem with a compatible MPI inside the container and the containerized MPI code to instantiate the job.
Currently, the only MPI library on Athena that supports the HPE Cray Slingshot 11 interconnect, for best communication performance, is the proprietary Cray MPICH. In contrast, containers typically include open‑source MPICH, Intel MPI, or OpenMPI. This article provides information on how to use Cray MPICH on the Athena host to work with MPICH, Intel MPI, or OpenMPI inside a container for multi-node runs.
To run your application, use a method that allows Cray MPICH to work with executables—either on the host or within a container—built with non-Cray-MPICH. As described in this article for MPI applications running natively on Athena, you can use one of the following two methods, depending on which flavor of MPI was used when the application was compiled:
If you have an application that was built with standard MPICH or Intel MPI—which are Application Binary Interface (ABI)-compatible with Cray MPICH—use the cray-mpich-abi module.
If you have a C/C++ (not Fortran) application that was built with OpenMPI, use the Cray MPICH ABI runtime translator, mpixlate, to translate the executable from OpenMPI to Cray MPI.
You will also use these methods later, when running an MPI application inside a container with a hybrid-MPI mode. An example is shown below in the sample script section.
Note: Starting June 15, 2026, a libfabric module is no longer loaded upon login. Instead, the libfabric library under /usr/lib64 will be used by default. However, for jobs that run with containers using the methods described in this article, you will need to load an hpe-libfabric module where the libfabric library is installed under /nasa instead of /usr/lib64 which causes collision with the container's /usr/lib64.
Step Two
Choose an appropriate Cray MPICH module that complies with the MPI standard supported by the MPI inside the container. Note that versions of MPI standard approved by the MPI Forum include 1.0, 1.1, 1.2, 1.3, 2.0, 2.1, 2.2, 3.0, 3.1, 4.0, 4.1 and 5.0.
Note: Currently, there are two Cray MPICH modules included in the Cray Programming Environment (CPE) 25.09 on Athena. If Athena is updated with a new CPE version, the Cray MPICH modules may change.
The default cray-mpich/8.1.33 (default), based upon Argonne National Laboratory (ANL) MPICH 3.4a2, provides full support of the MPI-3.1 standard
The cray-mpich/9.0.1, based on ANL MPICH 4.1.2, supports the MPI-4.0 standard
Be sure to load the appropriate Cray MPICH module depending on which MPI standard is required to support your application. For example, if the MPI library inside the container uses the MPI-4.0 standard, you will need to load the module cray-mpich/9.0.1 on the host.
Different flavors of MPI libraries available for public use are listed here:
Open-source MPICH
The version number of MPICH more or less reflects the version of MPI standard it supports.
MPICH 3.0.x supports MPI-3.0
MPICH 3.[1-4].x supports MPI-3.1
MPICH 4.0.x supports MPI-4.0
MPICH 4.[1-4].x supports MPI-4.1
MPICH 5.0.x supports MPI-5.0
MPICH includes the mpichversion command to easily check the version number. When using mpichversion from a non-default version of cray-mpich, you need to prepend $CRAY_LD_LIBRARY_PATH (defined in the corresponding cray-mpich module) to $LD_LIBRARY_PATH.
For example, if cray-mpich/8.1.33 is the default on a system, and you want to use cray-mpich/9.0.1 whose modulefile includes the following prepend-path:
then you must modify $LD_LIBRARY_PATH as demonstrated below:
host% module switch cray-mpich cray-mpich/9.0.1
host% which mpichversion
/opt/cray/pe/mpich/9.0.1/ofi/cray/20.0/bin/mpichversion
host% mpichversion
MPI VERSION : CRAY MPICH version 8.1.33.496 (ANL base 3.4a2)
^^ wrong version for cray-mpich/9.0.1 ^^
MPI BUILD INFO : Tue Jul 15 20:51 2025 (git hash b449595)
With bash,
host% export LD_LIBRARY_PATH="${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
With csh,
host% setenv LD_LIBRARY_PATH "${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
host% mpichversion
MPI VERSION : CRAY MPICH version 9.0.1.498 (ANL base 4.1.2)
^^ correct version for cray-mpich/9.0.1 ^^
MPI BUILD INFO : Wed Jul 16 9:35 2025 (git hash 0848216)
Intel MPI
Here are the Intel MPI versions and the MPI standard they support:
Versions 2017 and later support MPI-3.1 standard
Versions 2021.12 (distributed with oneAPI 2024.1) and later include support for MPI-4.0 features
Versions 2021.16 (distributed with oneAPI 2025.2) and later provides complete support for MPI-4.1
Version 2021.17 (distributed with oneAPI 2025.3) includes MPI-5.0 standard technical preview for C
OpenMPI
The OpenMPI version number does not reflect the MPI standard version it supports. For example,
MPI-3.0 standard conformance was introduced since OpenMPI 1.8
MPI-3.1 standard conformance was introduced since OpenMPI 2.0
Partial support for MPI-4.0 started with OpenMPI 5.x
TIP: Use the command ompi_info to find the version of MPI API it fully supports.
Step Three
Ensure that the host’s communication libraries are visible inside the container. This allows MPI jobs in the container to access Athena’s hardware and take advantage of its high-performance network and multiple network related libraries.
Set the Environment Variable APPTAINERENV_LD_LIBRARY_PATH on the Host
Use APPTAINERENV_LD_LIBRARY_PATH to set or override LD_LIBRARY_PATH inside the container. Ensure that the Libfabric library installed under the host's /nasa directory and the Cray MPICH, Process Management Interface (PMI) and Parallel Application Launch Service (PALS) libraries from the host's /opt/cray directory are defined in the container's library search path. If using the mpixlate utility, also add the path to the mpixlate library.
The examples below show how to set this variable on the host. The examples assume that Athena uses CPE 25.09, so the available Cray MPICH versions are 8.1.33 and 9.0.1. For the path_to_libfabric_library shown in the examples, use the LD_LIBRARY_PATH output from the command module show hpe-libfabric.
WARNING: The command lines shown in each section below are too long to be formatted as one line, so they are broken with a backslash (\). You should revert them back to a single long line in your script. There should be no space between entries of various libraries.
Example 1: If a container was built with the GNU compiler and an open-source MPICH library that supports the MPI-3.1 standard:
If other libraries are needed inside the container, add them just before /.singularity.d/libs.
Set the environment variable APPTAINER_CONTAINLIBS on the host
You can add the APPTAINER_CONTAINLIBS variable to specify a string of library file names (comma separated) to bind to the /.singularity.d/libs directory inside the container. This method is useful when a host library under (for example) /usr/lib64 is needed by the container, but binding the host /usr/lib64 to the container will collide with the container's /usr/lib64.
For example, the /usr/lib64/libcxi.so.1 library on the host is needed in order for a container to utilize the Cray Slingshot 11 interconnect. To make this library available to the container (and other libraries that the application may need), you can set:
If you experience errors indicating you are missing some additional libraries, check whether they exist under /usr/lib64 on the host and add them to the list above.
Step Four
For large message sizes, you might need to set the MPICH_SMP_SINGLE_COPY_MODE variable to NONE. This variable controls the intranode communication mechanism in HPE Cray MPICH (and related Cray/TOSS MPI builds), specifically for large message transfers between CPU processes. If you encounter hangs or errors, you can try debugging by setting this variable to NONE to disable high-performance single-copy shared memory optimizations, forcing data movement through slower paths.
There is more than one way to set this variable to pass it into the container. Here is one option:
If needed, use the NAS mbind.x tool—which does process/thread pinning—to bypass a known Cray PALS issue in which Cray MPICH copies the executable to /run/palsd/<some_hash>/executable to run multi-node jobs. This breaks codes that query the location of the executable to find files needed by the executable.
To bypass this issue, add mbind.x before the executable. The daemon palsd then copies mbind.x to /var/run/palsd/<some-hash>/mbind.x and runs the executable from its intended location.
When running an Apptainer container, the orginal executable is apptainer. In multi‑node runs, a failure can occur where palsd copies this executable (apptainer) from its installed location under /nasa to /run/palsd/<some_hash>/apptainer on the compute node. During startup, apptainer then searches for apptainer.conf but cannot find it as /run/palsd/<some_hash>../etc/apptainer/apptainer.conf, causing the run to fail. To work around this issue, replace:
Bind-mount the /opt/cray, /run/palsd, /nobackup/username, and /nasa filesystems for use by the container. Specifically, add the following after the apptainer exec command:
-B /opt/cray:/opt/cray
A container may already include the /opt directory. However, unless the container was built with the licensed Cray software stack, the /opt/cray directory should not exist within the container. As stated earlier, many of the communication libraries on the host under /opt/cray are needed for running a container in hybrid MPI mode. Therefore, you should bind mount /opt/cray for use by the container.
-B /run/palsd:/run/palsd
The palsd is the daemon of Cray's PALS for managing, launching, and coordinating MPI processes. The/run/palsd directory is used by PALS as a temporary runtime directory for socket files and communication metadata. It is needed when running the container across nodes.
-B /nobackup/username:/nobackup/username
This is needed if your container or anything you need to run the container is located under your /nobackup directory.
-B /nasa:/nasa
This is needed since the command apptainer and the libfabric library from an hpe-libfabric module that you load are is installed under /nasa on Athena.
Note: Your NAS $HOME filesystem is automatically bind mounted into the container as /u/username, not /homeX/username.
Sample Script
The sample script below is for the following scenario:
Running a container across two Athena compute nodes with 256 MPI ranks running on each node
Using an open-source MPICH library which supports MPI-3.1 standard inside the container
An executable inside the container built with GNU and MPICH
For other scenarios, modify the script with information provided in the Recipe section.
TIP: Test your setup with a hello world MPI before trying your executable.
#PBS -lselect=2:ncpus=256:mpiprocs=256:model=tur_ath
#PBS -lwalltime=2:00:00
#PBS -q normal
#PBS -j oe
#PBS -koed
#
# Note: By default, cray-mpich/8.1.33 and cray-pals/1.7.1
# should have been automatically loaded when your job starts.
# Load an hpe-libfabric module where libfabric is installed under /nasa
# instead of using libfabric under /usr/lib64
module switch PrgEnv-cray PrgEnv-gnu
module load hpe-libfabric
module load apptainer-x86-toss5
which mpiexec
which apptainer
# replace setenv xxx yyy with export xxx=yyy if
# using bash shell
# For APPTAINERENV_LD_LIBRARY_PATH,
# replace "path_to_libfabric_library" with the one
# shown in the output of module show hpe-libfabric
# If you encounter an error complaining that libpmi
# is not found, add /opt/cray/pe/pmi/default/lib
# Replace "other_libraries_from_container" with
# the actual library names inside container
# The two setenv command lines below are too long to be formatted as one line,
# so they are broken with a backslash (\). You should revert them back to a single long
# line in your script. There should be no space between entries of various libraries.
setenv APPTAINERENV_LD_LIBRARY_PATH \
"path_to_libfabric_library:/opt/cray/pe/mpich/8.1.33/ofi/gnu/11.2/ \
lib-abi-mpich:/opt/cray/pe/lib64:/opt/cray/pals/1.7/ \
lib:other_librarys_from_container:/.singularity.d/libs"
#It is possible that more libraries under /usr/lib64 are needed beyond the list below
setenv APPTAINER_CONTAINLIBS "/usr/lib64/libcxi.so.1,/usr/lib64/libjson-c.so.5, \
/usr/lib64/libnl-3.so.200,/usr/lib64/libtinfo.so.6"
mpiexec -l -np 512 --ppn 256 \
--env MPICH_SMP_SINGLE_COPY_MODE NONE \
--cpu-bind none /u/scicon/tools/bin/mbind.x \
apptainer exec \
-B /nobackup/username:/nobackup/username \
-B /opt/cray:/opt/cray \
-B /run/palsd:/run/palsd \
-B /nasa:/nasa \
/path/to/container_sif_file_or_sandbox \
/path/to/executable_inside_container
