In the Using Cray Message Passing Toolkit on Athena article, a method is described that allows applications built with Intel MPI to run on Athena, and take advantage of the Cray Slingshot 11 interconnect through the use of the cray-mpich-abi and cray-pals modules. In this article, we describe an alternative approach that doesn't use Cray MPICH while still benefiting from the high-performance Cray Slingshot 11 interconnect.

Specifically, this approach relies on setting a few Intel MPI environment variables to use Cray's Libfabric, PMI2 libraries, and the OpenFabrics Interface (OFI) CXI provider, and launching with the Cray Parallel Application Launch Service (PALS).

More information is included in the comments of the following sample script.

Note: The sample script below assumes that you're using the csh shell. For bash, replace the setenv command syntax with the export command syntax, as follows:

#for csh: 
setenv variable_name value

#for bash:
export variable_name=value

Sample Script

## This example runs an Intel MPI app on 2 nodes with 256 processes each.
#PBS -lselect=2:ncpus=256:mpiprocs=256:model=tur_ath
#PBS -lwalltime=2:00:00
#PBS -q normal
#PBS -j oe
#PBS -koed

## The executable will be launched with mpiexec from Cray PALS instead
## of from Intel MPI. This is due to the need to satisfy
## Virtual Network Interface (VNI) enforcement and security. 
## Load Intel MPI before Cray PALS to ensure CRAY PALS mpiexec is used.

## If any, also load other modules needed by the application.

module use -a /nasa/modulefiles/testing
module purge
module load mpi-intel/2021.16
module load cray-pals  

## Check that the mpiexec is from Cray Pals, not Intel MPI.

which mpiexec
mpiexec --version

## Use Cray's PMI2 instead of PMI to allow proper interaction with Intel MPI.

setenv I_MPI_PMI pmi2  
setenv I_MPI_PMI_LIBRARY /opt/cray/pe/pmi/default/lib/libpmi2.so  

## The Open Fabrics Interfaces (OFI) cxi provider is not distributed 
## in the libfabric of Intel MPI 2021.17 and earlier versions. 
## Direct Intel MPI not to use libfabric from Intel MPI and instead use 
## the libfabric library from Cray and use the cxi provider.

setenv I_MPI_OFI_LIBRARY_INTERNAL 0
setenv I_MPI_OFI_LIBRARY /opt/cray/libfabric/2.2.0rc1/lib64/libfabric.so.1
setenv I_MPI_OFI_PROVIDER cxi

## (Optional) Direct Intel MPI to check the OFI provider used.

#setenv I_MPI_DEBUG 1

## Direct Cray PALS mpiexec to use the PMI2 interface.

mpiexec -np 512 --ppn 256 \
--pmi=pmi2   \    
executable_built_with_intel_mpi