Building Applications

To build an MPI/OpenMP hybrid executable using HPE's MPT and Intel's OpenMP libraries, you must compile your code with the -openmp flag and link it with the -lmpi flag. For example:

%module load comp-intel/ mpi-hpe/mpt
%ifort -o your_executable prog.f -openmp -lmpi

Running Applications

Here is a sample PBS script for running an MPI/OpenMP application using 3 Skylake nodes, with 20 MPI processes on each node and 2 OpenMP threads per MPI process.

#PBS -lselect=3:ncpus=40:mpiprocs=20:model=sky_ele
#PBS -lwalltime=1:00:00

module load comp-intel/ mpi-hpe/mpt
setenv OMP_NUM_THREADS 2

cd $PBS_O_WORKDIR

mpiexec ./your_executable

You can specify the number of threads on the PBS resource request line using ompthreads. This causes the PBS prologue to set the OMP_NUM_THREADS environment variable.

#PBS -lselect=3:ncpus=40:mpiprocs=20::ompthreads=2:model=sky_ele
#PBS -lwalltime=1:00:00

module load comp-intel/ mpi-hpe/mpt

cd $PBS_O_WORKDIR

mpiexec ./your_executable

Performance Issues

For pure MPI codes built with HPE's MPT library, you can improve performance on Broadwell, Skylake, Cascade Lake, Rome, and Milan nodes by pinning the processes through setting MPI_DSM_DISTRIBUTE environment variables to 1 (or true). However, for MPI/OpenMP codes, all the OpenMP threads for the same MPI process have the same process ID. In this case, setting this variable to 1 causes all OpenMP threads to be pinned on the same core and the performance suffers.

It is recommended that you set MPI_DSM_DISTRIBUTE to 0 and use omplace for pinning instead.

If you use Intel version 10.1.015 or later, you should also set KMP_AFFINITY to disabled or OMPLACE_AFFINITY_COMPAT to ON as Intel's thread affinity interface would interfere with dplace and omplace.

#PBS -lselect=3:ncpus=40:mpiprocs=20::ompthreads=2:model=sky_ele
#PBS -lwalltime=1:00:00

module load comp-intel/ mpi-hpe/mpt

setenv MPI_DSM_DISTRIBUTE 0
setenv KMP_AFFINITY disabled

cd $PBS_O_WORKDIR

mpiexec -np 60 omplace ./your_executable