HPC-X Message Passing Interface (MPI) is an implementation of Open MPI (based on version 4.x) that includes features and optimizations from Mellanox, the vendor of the InfiniBand networking hardware used in HECC systems. HPC-X MPI offers improved performance for some MPI routines, however, the main purpose we are supporting this implementation is to provide you with an alternative to using the HPE Message Passing Toolkit (MPT).

Some MPI constructs that had been removed from the MPI 3.0 standard—but had remained available for use in deprecated status—were formally removed from OpenMPI-4.x. Our HPC-X MPI installation was built with a flag that makes these deprecated features visible again. For a list of these features, see Removed MPI Constructs on the Open MPI website.

Recommendation: Keep your MPI applications up to date with the current MPI standard.

To use HPC-X MPI on HECC systems, simply load the HPC-X module and your preferred Intel compiler suite, then recompile your application and its dependencies. Launch the newly built application as usual using the mpiexec launcher.

You can safely use the HPC-X build with any of the Intel compiler versions listed using command module avail comp-intel. The following example uses a specific Intel compiler:

% module purge
% module load comp-intel/2018.3.222 mpi-hpcx/2.4.0

The default process placement strategy is to place ranks round-robin by socket. This approach balances the computational load on the available hardware, but it might not be appropriate for all applications. You can force placement of ranks in linear core order by using the mpiexec --map-by option. For example:

% mpiexec -np XX --map-by ppr:YY:socket --bind-to core

where XX is the total number of ranks, YY is the user-specified number of ranks per socket, and the bind-to option is used to pin the ranks to specific cores.

Some workflow changes will be required if you manipulate $PBS_NODEFILE in order to launch multiple MPI jobs within a single PBS job. In this case, do the following:

• In the PBS resource request, include the maximum number of MPI ranks you intend to launch on each node.
• Create a separate rankfile for each invocation of mpiexec, using the following rankfile format:

  % rank XX=machineName slot=cpuID
  

  where XX is the MPI rank within the MPI sub-job, machineName is one of the names taken from the set provided by PBS (run uniq $PBS_NODEFILE to obtain this set), and cpuID is a hardware core ID (for core numbering, see the diagrams for each processor type: Cascade Lake, Skylake, Broadwell, Haswell, Ivy Bridge, Sandy Bridge).

This sample rankfile runs 16 ranks (0-15) evenly distributed on four sockets on two Ivy Bridge nodes (r469i0n10 and r469i0n11):

rank 0=r469i0n10 slot=0
rank 1=r469i0n10 slot=1
rank 2=r469i0n10 slot=2
rank 3=r469i0n10 slot=3
rank 4=r469i0n10 slot=10
rank 5=r469i0n10 slot=11
rank 6=r469i0n10 slot=12
rank 7=r469i0n10 slot=13
rank 8=r469i0n11 slot=0
rank 9=r469i0n11 slot=1
rank 10=r469i0n11 slot=2
rank 11=r469i0n11 slot=3
rank 12=r469i0n11 slot=10
rank 13=r469i0n11 slot=11
rank 14=r469i0n11 slot=12
rank 15=r469i0n11 slot=13