• Some OpenMPI modules installed by NAS staff are under /nasa/modulefiles/testing. Add this to your modulefiles search path in order to see all OpenMPI modules.

  % module use -a /nasa/modulefiles/testing
  % module avail nvhpc
  % module avail openmpi
  

  Note: NVIDIA provides version 3, 4, and HPCX-enhanced OpenMPI under its HPC SDK distributions. At NAS, they are installed under /nasa/nvidia/hpc_sdk/.../sdk_version_number/comm_libs/., where sdk_version_number is a number such as 23.11. When you load a modulefile such as nvhpc/23.11, an OpenMPI library is included. To find out which version of OpenMPI is used after you load a nvhpc modulefile, do, for example:

• module load nvhpc/23.11
  which mpiexec
  mpiexec --version
  

  The HPCX-enhanced OpenMPI is usually not set as the default. However, you may find the HPCX related modulefiles under:
  /nasa/nvidia/hpc_sdk/.../sdk_version_number/comm_libs/
       CUDA_version_number/hpcx/latest/modulefiles,

  where sdk_version_number is a number such as 23.11, and CUDA_version_number is a number such as 12.3.

• Use the command ompi_info to learn how an OpenMPI module was built. For example, what version of OpenMPI (3.x, 4.x, 5.x), flavor of the architecture (x86_64, aarch64), what C, C++, Fortran compilers (GNU, Intel, NVIDIA, LLVM), and configure options were used to build it.

  % module load an_openmpi_module
  % which ompi_info
  % ompi_info  (or ompi_info -all)
  

  Note: In newer releases (23.5 or newer) of OpenMPI 4.x or 4.x + hpcx included in the NVIDIA HPC SDK, the utility ompi_info may not be included in the installation.

Compile Time Tips

• OpenMPI provides the wrapper scripts mpicc, mpic++ (or mpicxx, mpiCC) and mpifort for compiling and linking MPI applications. The legacy wrappers mpif77 and mpif90 still exist but are symbolic links to mpifort. The compilers used by these wrappers default to the compilers used to build the OpenMPI library. Besides using the command ompi_info, one can also find the compilers used to build the MPI library by adding the option --showme when invoking the wrapper scripts. That is:

  % mpicc --showme
  % mpic++ --showme
  % mpifort --showme
  

  Note: OpenMPI in NVIDIA HPC SDK was built by NVIDIA with their nvc, nvc++ and nvfortran compilers.

• When building an MPI application written in C, most likely you can use an OpenMPI library built with any C compiler. For example, if the OpenMPI was built with gcc but you want to build your MPI application with clang, you can re-define the variable OMPI_CC and use the mpicc --showme command to check that the desired C compiler will be used as shown in the example below:

  % module load comp-aocc/4.2.0
  % module load openmpi/4.1.6-toss4-gnu
  % export OMPI_CC=clang
  % mpicc --showme
  clang -I.....
  % mpicc [options] program.c
  

• For C++ or Fortran MPI applications, the same compiler should be used for building both the MPI library and the application. This is discussed in OpenMPI documentation, section 4.12.4. Installing Multiple Copies of OpenMPI.

Runtime Tips

• If the OpenMPI library used was not successfully built with --with-tm to support PBS, a cross-node MPI job will need the mpiexec flag --hostfile $PBS_NODEFILE or --machinefile $PBS_NODEFILE during runtime. That is,

  mpiexec -np N --hostfile $PBS_NODEFILE ./a.out
  or
  mpiexec -np N --machinefile $PBS_NODEFILE ./a.out
  

  Another option is to use the --host flag as explained here.

• Before running your MPI application with an OpenMPI module, do a few basic tests as recommended by the OpenMPI developers in Part 3 of the ABCs of OpenMPI.
  • Single-node tests
    • To test the basic OpenMPI runtime system:

      mpirun -np 1 hostname
      

    • To test the start-up and shut-down of the MPI layer:

      mpirun -np 1 hello_c
      

    • To test the MPI communication with more than 1 process:

      mpirun -np 2 ring_c
      

  • Cross-node tests
    • To test the basic OpenMPI runtime system:

      mpirun -np 3 --host host1:1,host2:1,host3:1  hostname
      

    • To test the start-up and shut-down of the MPI layer:

      mpirun -np 3 --host host1:1,host2:1,host3:1 hello_c
      

    • To test the MPI communication with more than 1 process:

      mpirun -np 3 --host host1:1,host2:1,host3:1 ring_c
      

  Note: hostname is a Linux command. You can find the source code of hello_c and ring_c in github. host1, host2, host3 refer to the actual hostnames, for example, r705c1t1n[1-3] assigned to a PBS job.

• The mpiexec flag -x PATH -x LD_LIBRARY_PATH may be needed in some situations. For example, when combining OpenMPI with other tools such as the NAS mpiprof profiling tool for running cross-node jobs, use:

  module load scicon/mpiprof
  mpiexec -np N --hostfile $PBS_NODEFILE -x PATH -x \
       LD_LIBRARY_PATH mpiprof [mpiprof-options] ./a.out

  Note: The mpiexec command line is shown on two lines due to formatting issues. It should be all on one line.

• OpenMPI provides some process/thread binding capabilities through the use of a three-phase procedure using the flags --map-by <arg>, --rank-by <arg> and --bind-to <arg>.
  • mapping: Assigns a default location to each process.

    The argument for --map-by can be slot, hwthread, core, socket (default), numa, board, node.

  • ranking: Assigns an MPI_COMM_WORKD rank value to each process

    The argument for --rank-by can be slot (default), hwthread, core, socket, numa, board, node.

  • binding: Constrains each process to run on specific processors.

    For --bind-to, the possible arguments are none, hwthread, core, l1cache, l2cache, l3cache, socket, numa, board, cpu-list. Among them, none is the default when oversubscribed, core is the default when np <= 2, and socket is the default when np > 2.

  In addition, OpenMPI provides the flag --report-bindings to visualize bindings.

  Experiment with the --map-by, --rank-by, --bind-to, and --report-bindings options to get a sense of how these options work.

  For instance, running a.out with N=4 on 1 sky_ele node (2 sockets, each with 20 physical cores, or 40 logical cores when Hyperthreading is turned on in BIOS),

  • --map-by socket --rank-by core --bind-to core --report-bindings yields:

    rank 0 bound to socket 0 [core 0[hwt 0-1]]
    rank 1 bound to socket 0 [core 1[hwt 0-1] ]
    rank 2 bound to socket 1 [core 20[hwt 0-1]]
    rank 3 bound to socket 1 [core 21[hwt 0-1]]
    

  • --map-by core --rank-by core --bind-to core --report-bindings yields:

    rank 0 bound to socket 0 [core 0[hwt 0-1]]
    rank 1 bound to socket 0 [core 1[hwt 0-1]]
    rank 2 bound to socket 0 [core 2[hwt 0-1]]
    rank 3 bound to socket 0 [core 3[hwt 0-1]]

  You can also use the NAS inhouse mbind.x tool for binding. For example, to use mbind.x to spread the MPI processes among the physical cores in each 2-socket node, do:

  mpiexec -np N [mpiexec-options] /u/scicon/tools/bin/mbind.x -cs ./a.out
  

  This yields the following binding scheme with N=4 on 1 sky_ele node:

  host: xxx, ncpus: 80, rank: 0 (r0), bound to cpu: 0
  host: xxx, ncpus: 80, rank: 1 (r1), bound to cpu: 10
  host: xxx, ncpus: 80, rank: 2 (r2), bound to cpu: 20
  host: xxx, ncpus: 80, rank: 3 (r3), bound to cpu: 30
  

  Note: The mbind.x tool developer recommends to use the OpenMPI's own binding capabilities instead of mbind.x in case of bugs or incompatibilities.

• For better control of the number of MPI ranks to run on each node, use one of the choices below. This is useful in situations where there are more slots availble in each node than what you want to use. For example, in a batch job with 2 nodes each with 40 cores, if you want to run just 8 MPI ranks across 2 nodes each with 4 MPI ranks, do:
  • Use the --host or -H option: (all one line)
    mpirun -np 8 --host host1:4,host2:4 --hostfile $PBS_NODEFILE -x
        PATH -x LD_LIBRARY_PATH ./a.out

    where host1 and host2 are actual hostnames such as r101i5n0 and r101i5n1. You can find the hostnames of nodes assigned to your job by:
    cat $PBS_NODEFILE | uniq

  • Use the -npernode option (all one line):
    mpirun -np 8 -npernode 4 --hostfile $PBS_NODEFILE -x
        PATH -x LD_LIBRARY_PATH ./a.out    
  • Use the -N option  (all one line):
    mpirun -np 8 -N 4 --hostfile $PBS_NODEFILE -x
        PATH -x LD_LIBRARY_PATH ./a.out   
  • Use the --map-by option  (all one line):
    mpirun -np 8 --map-by ppr:4:node --hostfile $PBS_NODEFILE -x
        PATH -x LD_LIBRARY_PATH ./a.out    
• The inter-node network of NAS systems mainly use InfiniBand. The OpenMPI modules installed by NAS staff should have included support for InfiniBand. For example, if the OpenMPI was built with UCX, then UCX should automatically detect the presence of InfiniBand and will use it by default. If it was not built with UCX, then likely, it was built with support of openib in the btl framework. In this case, openib will also detect the presence of InfiniBand.

  Although not recommended, you can still control whether or how to use InfiniBand by adding --mca flags. For example,

  mpiexec -np N --mca pml ucx --mca btl ^openib ./a.out
  mpiexec -np N --mca btl self,vader,tcp ./a.out
  

  Note: ^openib means not to use openib for communication. TCP (Transmission Control Protocol) is a communication standard that allows devices to exchange messages over a network. Performance using TCP is slower than using InfiniBand RDMA (remote direct memory access).

• Starting in OpenMPI 5.0.0, you can view which networks (such as sm, uct) are used during runtime between different pairs of nodes by adding the flag --mca ompi_display_comm 1. That is,

  mpiexec -np N --hostfile $PBS_NODEFILE --mca_display_comm 1 ./a.out
  

  This will generate a communication matrix to stdout.

• If your application is built with a recent OpenMPI version that uses UCX and experiences large run-to-run variations and/or degraded performance in communication time, compared to an older OpenMPI version or other MPI implementations (for example, HPE MPT), consider adding the UCX environment variables UCX_PROTO_ENABLE=n and UCX_GVA_ENABLE=n as follows:

  mpiexec -np .. -x UCX_PROTO_ENABLE=n -x UCX_GVA_ENABLE=n ...

  Adding these two variables can help applications including LAMMPS, NPB FT, and IS benchmarks.

  For NPB FT, adding another variable UCX_IB_MLX5_DEVX=no also helps.

  For more information, read Known Issues: 3606445 (discovered in HPCX 2.17.0) and 3884209 (discovered in HPCX 2.19.0) on the use of UCX_PROTO_ENABLE=n; 4050321 (discovered in HPCX 2.21.0) on the use of UCX_GVA_ENABLE=n; and 4026461 (discovered in HPCX 2.20.0) on the use of UCX_IB_MLX5_DEVX=no.

• To explicitly choose which MPI IO implementation to use, add the flag --mca io .... For example,

  mpiexec -np N --mca io ompio ./a.out
  or
  mpiexec -np N --mca io romio314 ./a.out
  

References

• OpenMPI: Open Source High Performance Computing
• Slides and Videos of "What is MPI? What is OpenMP?" and "The ABCs of OpenMPI, parts 1 - 3"
• Open-MPI-SC23-BOF.pdf