Running Apptainer in hybrid MPI mode allows you to call mpiexec or a similar launcher on the apptainer command itself, when you execute an Apptainer container with an MPI code. The MPI process outside of the container will then work in tandem with MPI inside the container and the containerized MPI code to instantiate the job.

Apptainer supports two main open-source implementations of MPI: OpenMPI and MPICH. This article provides information primarily on using OpenMPI 3.x and 4.x to run on some NAS systems, for example, Cabeus, Aitken, and Electra.

How OpenMPI Interacts with Apptainer in Hybrid Mode

The workflow described on the Apptainer website is duplicated here:

1. The MPI launcher (such as mpirun, mpiexec) is called by the resource manager or the user directly from a shell.
2. OpenMPI then calls the process management daemon (orted).
3. The orted process launches the Apptainer container requested by the launcher command.
4. Apptainer instantiates the container and namespace environment.
5. Apptainer then launches the MPI application within the container.
6. The MPI application launches and loads the OpenMPI libraries.
7. The OpenMPI libraries connect back to the orted process via the Process Management Interface (PMI).

At this point, the processes within the container run as they would normally, directly on the host.

Important: The MPI in the container must be compatible with the version of MPI available on the host. For best performance, the container’s MPI should be configured for optimal use of the system hardware. The standard approach is to build a matching OpenMPI version on the host, as described in the next section.

Build Your Own OpenMPI on the Host

NAS systems typically do not provide an OpenMPI version compatible with your container. To build a compatible OpenMPI on the host, determine the container’s MPI version and build configuration by running the ompi_info command inside the container. The following example shows OpenMPI version (4.1.2) and the configure options used inside a container.

Note: If your container includes an OpenMPI with version 5.x, you will need a compatible OpenMPI 5.x on the host.

nas_host% apptainer shell your_container.sif (or  your_container_sandbox )
Apptainer> ompi_info
                Open MPI: 4.1.2
...
  Configure command line: '--enable-orterun-prefix-by-default' '--with-verbs'
                          '--prefix=/usr/local/mpi'
...

To build your own OpenMPI on the host:

1. Download an OpenMPI software version that is the same as, or newer than, the version used inside the container.
2. Install the software in your /nobackup filesystem by using --prefix=/nobackuppX/username/...

   Note: NAS users do not have permission to install OpenMPI in any system location, including the /usr/local directory.

3. Configure the OpenMPI with options similar to those used inside the container.

Notes:

• The --with-verbs option is typically unnecessary, because OpenMPI automatically detects and supports InfiniBand (IB) on NAS systems. (The --with-verbs option builds support for OpenFabrics verbs, previously known as "Open IB".)
• Do not include the --with-pmi option (where PMI means Process Management Interface) when building on NAS systems because PMI supports launching jobs with SLURM rather than PBS.

Test the Hybrid MPI Mode with a Simple MPI Code

Before running your MPI application in hybrid mode, compile a HelloWorldMPI code using the compiler and MPI library inside the container, and run it in hybrid mode. If the HelloWorldMPI code does not run properly, your MPI application won't either.

Note: If the apptainer exec -e option is used, it likely would result in every MPI rank acting as rank 0, as demonstrated in the following example, which uses a total of four MPI processes (from two nodes with two MPI processes on each node):

mpirun -np 4 -H r101i3n12:2,r101i3n13:2 \
apptainer exec -e your_container_sandbox $HOME/hello_world_mpi_built_inside_container
or
mpirun -np 4 -N 2 \
apptainer exec -e your_container_sandbox $HOME/hello_world_mpi_built_inside_container


Hello world from processor r101i3n12, rank 0 out of 1 processors
Hello world from processor r101i3n13, rank 0 out of 1 processors
Hello world from processor r101i3n12, rank 0 out of 1 processors
Hello world from processor r101i3n13, rank 0 out of 1 processors

Depending on the required setup—for example, removing the -e option and adding some MPI-related environment variables—a successful HelloWorldMPI run should produce:

Hello world from processor r101i3n12, rank 0 out of 4 processors
Hello world from processor r101i3n13, rank 2 out of 4 processors
Hello world from processor r101i3n12, rank 1 out of 4 processors
Hello world from processor r101i3n13, rank 3 out of 4 processors

Run the Application with OpenMPI in Hybrid Mode

The following is a more elaborate example showing how to run applications built with OpenMPI in hybrid MPI mode. A total of four MPI processes is used (from two compute nodes with two MPI processes on each node). The OpenMPI inside the container is installed in the /usr/local/mpi directory.

mpiexec -np 4 -H r101i3n12:2,r101i3n13:2 --mca btl vader,openib,self \
apptainer exec -B /nasa:/nasa -B /nobackuppX/username:/nobackuppX/username \
--env OPAL_PREFIX="/usr/local/mpi" --env PMIX_INSTALL_PREFIX="/usr/local/mpi" \
your_container_sandbox a.out
or
mpiexec -np 4 -N 2 --mca btl vader,openib,self \
apptainer exec -B /nasa:/nasa -B /nobackuppX/username:/nobackuppX/username \
--env OPAL_PREFIX="/usr/local/mpi" --env PMIX_INSTALL_PREFIX="/usr/local/mpi" \
your_container_sandbox a.out

• The -H option is used to properly list the compute nodes to invoke processes on (you will need to find the hostnames of these nodes, such as r101i3n12 and r101i3n13). Without the -H option, all of the processes would run on the first node, possibly oversubscribing the node.
• The -N option specifies the number of processes per host.
• If the host and container OpenMPI libraries were built to support openib, use ‑‑mca btl vader,openib,self to explicitly use the vader, openib, and self byte transfer layer (BTL) components of the modular component architecture (MCA) to transfer data between two processes.

  Read the next section to learn more about the BTL components.

• The -B option is used to bind mount the NAS /nobackup filesystem and the /nasa directory, which is necessary if any required software is installed in those locations (such as GCC and OpenMPI on the host).
• The apptainer exec --env option is used to properly set the OPAL_PREFIX and PMIX_INSTALL_PREFIX environment variables to the MPI installation inside the container and ensure they are seen by every MPI process on every node. This option may be needed to resolve conflicts between the host and the container.

  Alternatively, instead of using the --env option, you can define the variables APPTAINERENV_OPAL_PREFIX and APPTAINERENV_PMIX_INSTALL_PREFIX before invoking mpiexec:

  with bash:
  export APPTAINERENV_OPAL_PREFIX=/usr/local/mpi
  export APPTAINERENV_PMIX_INSTALL_PREFIX=/usr/local/mpi
  
  with csh:
  setenv APPTAINERENV_OPAL_PREFIX /usr/local/mpi
  setenv APPTAINERENV_PMIX_INSTALL_PREFIX /usr/local/mpi
  
  mpiexec -np 4 ...
  

  Note: On the host environment, the OPAL_PREFIX and PMIX_INSTALL_PREFIX variables still need to be set to the MPI installation on the host in order for mpiexec to work properly.

For GPUs: Read this article if you plan to run containers on NVIDIA GPUs.

More Information About the OpenMPI MCA BTL

You can see which BTL components are supported in your OpenMPI installation by running the ompi_info command. For example:

Apptainer> ompi_info | grep 'MCA btl'
             MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.1.2)
             MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.1.2)
             MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.1.2)
             MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.2)

These components are described below:

• The self component is used for loopback communication (i.e., when an MPI process sends to itself).
• The openib component uses OpenFabrics Alliance verbs to support InfiniBand, RDMA over Converged Ethernet (RoCE), and RDMA over Internet Protocol networks (iWARP).
• The tcp component uses TCP-based communications over IP interfaces and networks.
• The vader component uses shared memory.

You typically don’t need to specify BTL components on the mpirun command line, because OpenMPI automatically selects the best option for each communication. Manual selection is only useful when you want to use a specific BTL, or to override a default such as ^openib (which disables OpenIB) in your OpenMPI modulefile and you want to be sure to use openib to prevent slow TCP communication or possible timeouts in some containerized applications.

Notes:

• If you are specifying BTL components explicitly, be sure to include self. If the self component is not specified, OpenMPI may be unable to complete send-to-self scenarios.
• If using OpenMPI 5.x, the available --mca BTL components will be different. For example, vader is renamed to sm and openib is removed. You may also need to specify other --mca frameworks such as pml (point-to-point messaging layer), etc.

Check if You are Using a NAS-Instrumented mpirun or mpiexec

If you are using an NAS-built MPI modulefile on the host, the mpirun and mpiexec commands may be instrumented. The originals are available as mpirun.real and mpiexec.real.

If the instrumented mpirun and mpiexec on the host fail to run the container, try using mpirun.real or mpiexec.real instead.

References

• Apptainer and MPI Applications
• OpenMPI Frequently Asked Questions (for OpebMPI 4.x and earlier)