To help you prepare for running jobs on Aitken's Milan nodes, this short user guide includes information on the general configuration of Milan nodes, compiling your code, and running PBS jobs.

Aitken Milan Front-End Systems

Two new Aitken front ends, afe01 and afe02, serve the Aitken Milan nodes. Each AFE includes one or two AMD EPYC 7763 processors with a total of 128 physical cores (sum of two sockets), and 512 gigabytes (GB) of memory.

Note: The Aitken Milan compute nodes are not accessible from the PFEs.

You can access the AFEs using one of the following methods.

Initiating SSH from Your Local Desktop (Recommended)

• Two-step login:

  local_desktop% ssh sfe6.nas.nasa.gov  (or replace sfe6 with sfe7 or sfe8)
  sfe6% ssh afe01    (or ssh afe02) 

• One-step login (using SSH passthrough):

  Before you begin: Follow the instructions in Setting Up SSH Passthrough, and make sure that the configuration file in the .ssh directory of your local desktop contains the following lines:

  Host afe01 
      HostKeyAlias            afe01.nas.nasa.gov
      ProxyCommand            ssh -ax -oCompression=no sfe ssh-balance %h
  
  Host afe02 
      HostKeyAlias            afe02.nas.nasa.gov
      ProxyCommand            ssh -ax -oCompression=no sfe ssh-balance %h

  Once you have SSH Passthrough set up, you can log into the AFE in one step:

  local_desktop% ssh afe01   (or ssh afe02)

Initiating SSH from a PFE

Note: This method is not recommended but is made available for convenience.

pfe% ssh afe01 (or ssh afe02)

Aitken Milan Compute Nodes

Aitken includes 512 Milan compute nodes, which are partitioned into four physical racks. Each node contains two 64-core AMD EPYC 7763 Milan sockets (2.45 GHz) and 512 GB of memory.

These nodes are inter-connected with four-lane High Data Rate (4X HDR) devices and four-lane High Data Rate (4X HDR) switches. They are currently housed in the NAS main building and have access to the NAS compute resources home and Lustre filesystems.

Note: For the long term, the Milan nodes will be relocated to be near the existing Aitken resources when a new compute module becomes available. At that time, these nodes will be connected to the Rome nodes with additional HDR cables on the 8th dimension of the Aitken cluster hybercube.

Operating System and Software Modules

The operating system on all Milan compute nodes is the Red Hat Enterprise Linux-based Tri-Lab Operating System Stack Version 4 (TOSS 4). Batch jobs use :aoe=toss4 by default, so you do not need to add :aoe=toss4 in your PBS resource request. The default $MODULEPATH includes:

• /usr/share/Modules/modulefiles
• /nasa/modulefiles/toss4
• /nasa/modulefiles/spack/
• /nasa/modulefiles/pkgsrc/toss4/

Compilers For Milan Nodes

There are several compilers you can use to build your application:

Intel Compilers

Some Intel compiler modules are available in the /nasa/modulefiles/toss4 directory, however, several are in the /nasa/modulefiles/testing directory. To see what versions are available in these two directories, run:

module use -a /nasa/modulefiles/testing
module avail comp-intel

To build your application using the Intel compilers, use -march=core-avx2.

Existing executables that are built for the Aitken Rome nodes should run on the Milan nodes.

Note: Although AMD recommends using -axCORE-AVX2, we do not recommend using it, as the resulting executable will run but will not actually take advantage of AVX2 instructions. Executables built with -xCORE-AVX2, -xSSE4.2, and so on will not run on Milan processors.

GNU Compilers

The default version of GNU Compiler Collection (GCC) under TOSS 4 (/usr/bin/gcc) is GCC 8.5.0. However, support for the Rome architecture is only available with GCC 11.xx and later. GCC version 13.2 has been installed and is accessible via:

module use -a /nasa/modulefiles/testing
module load gcc/13.2

The recommended compiler flag for Milan with gcc is: -march=znver3

AMD Optimizing C/C++ Compilers (AOCC)

To see available versions before loading an AOCC compiler, use the module avail comp-aocc command. For example:

module use -a /nasa/modulefiles/testing
module avail comp-aocc
----------- /nasa/modulefiles/testing ----------
comp-aocc/4.0.0  comp-aocc/4.1.0  cmp-aocc/4.2.0  

----------- /nasa/modulefiles/toss4 ------------
comp-aocc/3.2.0  

The AOCC C, C++ and Fortran compiler drivers are clang, clang++ and flang, respectively.

The recommended compiler flag for Milan with AOCC is: -march=znver3

Note: The default optimization level is -O1 in AOCC 3.2, and -O2 in AOCC 4.1 and newer versions.

For more compiler options, see AMD's article Compiler Options Quick Reference Guide for AMD EPYC 7xx3 Series Processor.

MPI Libraries For Milan Nodes

HPE MPT

Use the modulefile mpi-hpe/mpt, which is available in the /nasa/modulefiles/toss4 directory:

module load mpi-hpe/mpt

OpenMPI

There are a few OpenMPI modulefiles available in the /nasa/modulefiles/testing directory.

module use -a /nasa/modulefiles/testing
module avail openmpi
---- /nasa/modulefiles/testing ----
openmpi/4.1.5-toss4-intel_2022.1  openmpi/4.1.6-toss4-gnu  openmpi/4.1.6-toss4-aocc 

Note: There are several newer versions that are not listed in the output above.

The compiler used to build each of these modulefiles is indicated in their filenames. For example, openmpi/4.1.6-toss4-gnu was built with gcc 8.5.0, and openmpi/4.1.6-toss4-aocc was built with aocc 4.2.0.

After loading one of these modulefiles, run the command ompi_info to learn more about each installation.

Running OpenMP or MPI/OpenMPI Hybrid Codes

Each Milan node has 128 physical cores. When simultaneous multi-threading (SMT) is enabled in BIOS, there are 256 logical cores for use by OpenMP or MPI/OpenMPI hybrid codes.

To check whether SMT is enabled, run the following command on a Milan compute node:

cat /proc/cpuinfo | grep processor | wc -l

If the output shows 256, then SMT is enabled.

Note: During Milan's early production, some nodes were configured with the SMT on and some with the SMT off. Currently, all nodes have been set with SMT on.

We recommend using the mbind.x tool for process and thread binding, as shown in the following example with four MPI processes and 32 OpenMP threads per process:

mpiexec -n 4 /u/scicon/tools/bin/mbind.x -t 32 a.out

If you use HPE MPT, you can also use the omplace tool for pinning MPI/OpenMP hybrid code.

Running PBS Jobs on Aitken Milan Nodes

Aitken's Milan nodes are accessible only through PBS jobs submitted from afe01 and afe02 and served by the PBS server pbspl1.

To request Aitken Milan nodes, use :model=mil_ait in your PBS script, as shown in the example below.

Sample PBS Script For Aitken Milan Nodes

#PBS -l select=2:ncpus=128:mpiprocs=128:model=mil_ait 
#PBS -l walltime=8:00:00 
#PBS -q normal 
module use -a /nasa/modulefiles/testing 
module load mpi-hpe/mpt
module load comp-intel
cd $PBS_O_WORKDIR 
mpiexec -np 256 ./a.out

For more information, see AMD Milan Processors.