To help you prepare for running jobs on Aitken's Rome nodes, this short user guide includes information on the general configuration of Rome nodes, compiling your code, and running PBS jobs.

Overview of Aitken Rome Nodes

Aitken includes 2,008 Rome nodes, which are partitioned into sixteen physical racks. Each node contains two 64-core AMD EPYC 7742 Rome sockets (2.25 GHz) and 512 GB of memory. The nodes are connected to the Aitken InfiniBand network (ib0 and ib1) via four-lane High Data Rate (4X HDR) devices and four-lane High Data Rate (4X HDR) switches for internode communication. The ib1 fabric is used mainly for I/O and is connected to the Lustre filesystems. In addition, Aitken Rome nodes share the same home filesystems, PFEs, and PBS server as Electra, and the Aitken Cascade Lake and Milan nodes. You can access Aitken Rome nodes only through PBS jobs submitted from the PFEs. 

Note: Aitken Milan nodes use a different set of front ends. See Preparing to Run on Aitken Milan Nodes for more information.

Operating System and Software Modules

The operating system on all Rome compute nodes is the Red Hat Enterprise Linux-based Tri-Lab Operating System Stack Version 4 (TOSS 4). Batch jobs use :aoe=toss4 by default, so you do not need to add :aoe=toss4 in your PBS resource request. The default $MODULEPATH includes:

• /usr/share/Modules/modulefiles
• /nasa/modulefiles/toss4
• /nasa/modulefiles/spack/
• /nasa/modulefiles/pkgsrc/toss4/

Compilers for Rome Nodes

Note: Some executables built under the TOSS 3 environment will work under TOSS 4; however, some may not. If you need to rebuild your executable under TOSS 4, and you experience issues such as missing certain software packages or libraries, send an email to support@nas.nasa.gov

There are several compilers you can use to build your application:

• Intel Compilers:

  Some Intel compiler modules are available in the /nasa/modulefiles/toss4 directory, however, several are in the /nasa/modulefiles/testing directory. To see what versions are available in these two directories, run:

  module use -a /nasa/modulefiles/testing
  module avail comp-intel
  

  Existing executables that are built with support for AVX512 (for running on Skylake and Cascade Lake processors) will not run on the Rome nodes. Others that are built with support for AVX2 may run on the Rome nodes; however, you should always check the result of your runs using existing executables. To rebuild your application using Intel compilers, use -march=core-avx2.

  Note: Although AMD recommends using -axCORE-AVX2, we do not recommend using it, as the resulting executable will run but will not actually take advantage of AVX2 instructions. Executables built with -xCORE-AVX2, -xSSE4.2, etc., will not run on Rome processors.
• GNU Compilers:

  The default version of GNU Compiler Collection (GCC) under TOSS 4 (/usr/bin/gcc) is GCC 8.5.0. However, support for the Rome architecture is only available with GCC 9.x and later. Among the several GCC versions available in the /nasa/pkgsrc/toss4 directory, load the 9.3 or 10.3 version for use with the Rome nodes. For example:

  module avail gcc
  -- /nasa/modulefiles/pkgsrc/toss4/ --
  gcc/6.5  gcc/7.5  gcc/8.4  gcc/9.3  gcc/10.3  gcc/13.2
  module load gcc/10.3

  The recommended compiler flag to use with GCC is -march=znver2.
• AMD Optimizing C/C++ and Fortran Compilers (AOCC)

  To see available versions before loading an AOCC compiler, use the module avail comp-aocc command. For example:

  module avail comp-aocc
  -- /nasa/modulefiles/toss4 --
  comp-aocc/3.2.0
  
  -- /nasa/modulefiles/testing --
  comp-aocc/4.0.0  comp-aocc/4.1.0  comp-aocc/4.2.0  comp-aocc/5.0.0
  
  module load comp-aocc/3.2.0

  The recommended compiler flag to use with AOCC is -march=znver2.
• PGI Compilers:

  Some PGI compiler modules are available in the /nasa/modulefiles/testing directory. There is no support specifically for Rome processors. If you want to compile your code using a PGI compiler, the recommended compiler flag is -tp=zen

For more compiler options, see Compiler Options Quick Reference Guide for AMD EPYC 7xx2 Series Processors.

MPI Library for Rome Nodes

HPE MPT is the only recommended MPI library for use on the AMD Rome nodes under TOSS 4. Use the mpi-hpe/mpt.2.30 modulefile in the /nasa/modulefiles/toss4 directory. Earlier versions such as mpi-hpe/mpt.2.25 do not work properly for the Rome nodes.

module load mpi‑hpe/mpt.2.30

Note: Although there is more than one version of MPT library installed on NAS systems, mpi‑hpe/mpt.2.30 is currently the only reliable version for running on the Rome nodes. You can access this version by loading mpi‑hpe/mpt, which points to the recommended version (currently mpi‑hpe/mpt.2.30) and environmental settings:

% module load mpi-hpe/mpt

Running OpenMP or MPI/OpenMPI Hybrid Codes

Each Rome node has 128 physical cores. When Simultaneous Multi-Threading (SMT) is enabled in BIOS, there are 256 logical cores for use by OpenMP or MPI/OpenMPI hybrid codes.

To check whether SMT is enabled, run the following command on a Rome compute node:

cat /proc/cpuinfo | grep processor | wc -l

If it shows 256, then SMT is enabled.

We recommend using the mbind.x tool for process and thread binding, as shown in the following example with four MPI processes and 32 OpenMP threads per process:

mpiexec -n 4 /u/scicon/tools/bin/mbind.x -t 32 a.out

If you use HPE MPT, you can also use the omplace tool for pinning MPI/OpenMP hybrid code.

Running PBS Jobs on Aitken Rome Nodes

To request Aitken Rome nodes, use :model=rom_ait in your PBS script, as shown in the example below.

The devel, normal, long, debug, and low queues are available for use with Rome nodes.

Note: The Standard Billing Unit (SBU) rate for Rome processors is 4.06.

Sample PBS Script For Aitken Rome Nodes

#PBS -l select=2:ncpus=128:mpiprocs=128:model=rom_ait
#PBS -l walltime=8:00:00
module load mpi-hpe/mpt
module load comp-intel/2020.4.304

cd $PBS_O_WORKDIR

mpiexec -np 256 ./a.out

For more information, see AMD Rome Processors.