The Cabeus cluster comprises multiple NVIDIA GPU nodes, including:

• 128 nodes with Milan CPU and NVIDIA A100 GPUs (model=mil_a100)
• 350 nodes with NVIDIA GH200 Superchip (model=gh200)

Among them, the gh200 nodes are served by their own front ends (cghfe01 and cghfe02) and PBS server (pbs05a), while the other GPU nodes are served by the front ends cfe01, cfe02 and PBS server pbspl4.

In this article, the term Cabeus A100 GPU nodes refers to the 128 nodes with Milan CPU and NVIDIA A100 GPUs.

To help you prepare for running jobs on the Cabeus A100 GPU nodes, this short user guide includes information on the general configuration of the nodes, compiling your code, and running PBS jobs. 

Overview of Cabeus Nodes with AMD EPYC CPU and NVIDIA A100 GPUs

The Cabeus cluster includes two Cabeus front-end nodes (cfe01 and cfe02) for the GPU nodes with x86_64-based CPUs (mil_a100). Each of the CFEs includes one 16-core AMD EPYC 7313p processor and 256 GB memory. 

Note: The Cabeus compute nodes are not accessible from the PFEs.

Each of the 128 A100 compute nodes, referred to as the mil_a100 nodes, contains:

• One 64-physical-core AMD EPYC 7763 Milan CPU socket (2.45 GHz) with 512 GB DDR4 host memory.
• Four NVIDIA A100 accelerator cards each with 3,456 double precision CUDA cores, 432 Tensor cores (boost clock 1410 MHz), and 80 GB HBM2e device memory
• Third-generation NVLink with 12 links yielding 600 GB/sec total bandwidth for GPU-to-GPU communication
• PCIe 4.0 bus connecting the CPU socket and the GPU accelerator cards with a bandwidth of 31.5 GB/s per direction

These 128 compute nodes are partitioned into twelve physical racks. They are connected in a fat-tree topology to the Cabeus InfiniBand network (ib0 and ib1) via four-lane High Data Rate (4X HDR) devices and 4X HDR switches. These HDR devices and switches provide a maximum bandwidth of 200 Gbits/s (i.e., 25 GB/s) for internode communication.

All existing HECC filesystems such as the home, VAST, and Lustre filesystems are mounted on both CFEs and on the Cabeus compute nodes.

PBS jobs for running on Cabeus A100 nodes are managed by the PBS server pbspl4.

Requesting GPU Allocations

An SBU allocation specifically for the GPUs is required to use the GPU resources. For more information about requesting GPU allocations, see this article.

The SBU rates for using one GPU node for one hour, for each type of node, are:

• mil_a100 with four A100s: 37.86
• gh200 with one GH200 Superchip: 18.34

Depending on how you request the GPU resources, your job may be charged the full rate or a fraction of the SBU rate. See the next section for some examples.

Accessing the CFEs and the mil_a100 nodes

You can access the Cabeus front ends (CFEs) either from your local desktop or from the PFEs. If you have set up SSH passthrough for the CFEs, then do:

your_localhost% ssh cfe01   (or ssh cfe02)

If you are connected to a PFE, then do:

pfe% ssh cfe01 (or ssh cfe02)

You can access the Cabeus Milan A100 compute nodes only through PBS jobs submitted from the CFEs or from pbspl4. To request Milan A100 (mil_a100) resources, specify one of the queue names shown below and the model type mil_a100.

The following PBS attributes are required when requesting the A100 resources:

• model=mil_a100: The A100 nodes with the AMD EPYC 7763 CPU are designated with a model name mil_a100.
• -q, followed by one of the queues shown below, to access the mil_a100 nodes.

The PBS queues for the Milan A100 nodes are as follows:

• gpu_normal 
  • Maximum wall time: 24 hrs
  • Maximum node count: 32
     
• gpu_devel
  • Maximum wall time: 2 hrs
  • Maximum node count: 2
  • Maximum user jobs (queued or running): 1
  • Priority: 10
     
• np_gpu_wide (non-primetime)
  • Maximum wall time: 12 hrs
  • Maximum node count: 64
  • Minimum node count: 24
  • Maximum user jobs (queued or running): 2
  • Priority: 20

Important: The np_gpu_wide queue will run jobs between 7:00 a.m. Thursdays and 7:00 a.m. Saturdays (Pacific Time). You can submit jobs to the np_gpu_wide queue at any time during the week, but the jobs will run only during this period.

As shown in the examples below, adding @pbspl4 to the queue name is optional, whether you submit the job from a CFE or from pbspl4.

cfe% qsub -I -q gpu_devel@pbspl4 
     -lselect=1:ncpus=10:ngpus=1:mem=xxGB:model=mil_a100

or

pbspl4% qsub -I -q gpu_devel 
     -lselect=1:ncpus=10:ngpus=1:mem=xxGB:model=mil_a100

Note: The qsub commands are shown on two lines due to a KB formatting issue. They should be on one line.

When not specified, the default value of ncpus, ngpus, and host mem for all GPU node types (including mil_a100) is 1, 1, and 32 GB, respectively. The minimum value of ncpus, ngpus, and host mem is 1, 1, and 256 MB, respectively.

From the perspective of PBS, one physical node can be made up of multiple virtual nodes (vnodes). In the case of mil_a100, one physical node with 4 GPUs is partitioned as 4 vnodes, where each vnode includes 1 GPU, 16 physical cores, and 128 GB of memory.

PBS allocates resources in the vnodes differently depending on the actual selection specified in -lselect=… and the choice of the PBS -lplace statement, which includes an arrangement attribute (free, pack, scatter, vscatter) and a sharing attribute (shared, excl, and exclhost). Read man qsub for the definition of each value of the arrangement and the sharing attribute.

If you plan to use fewer than 4 GPUs per physical mil_a100 node, you can use the shared sharing attribute which allows other jobs to use the remaining GPUs in the node. It also allows your job to be charged for only the number of GPUs assigned to your job. For example, the following request will be charged for the use of only 1 GPU in a mil_a100 node:

-lselect=1:ncpus=16:ngpus=1:mem=60GB:model=mil_a100 -lplace=scatter:shared

If you do not want other jobs to share resources in the mil_a100 nodes assigned to your job, it is easier if you simply request the use of all 4 GPUs in each mil_a100 node. For example:

-lselect=1:ncpus=16:ngpus=4:mem=60GB:model=mil_a100 -lplace=scatter:excl

This request will result in being charged for all 4 GPUs in a mil_a100 node.

Operating System and Software Modules

The operating system on all A100 compute nodes is TOSS 4 (Red Hat Enterprise Linux-based Tri-Lab Operating System Stack Version 4). Batch jobs use :aoe=toss4 by default, so you do not need to add :aoe=toss4 in your PBS resource request. The default $MODULEPATH includes:

• /usr/share/Modules/modulefiles
• /nasa/modulefiles/toss4
• /nasa/modulefiles/spack
• /nasa/modulefiles/pkgsrc/toss4

Programming and Compiling For Cabeus A100 Nodes

Read Programming for NVIDIA GPUs and the references listed there for information on the recommended programming models for developing applications suitable for running on the NVIDIA GPUs. 

Depending on the programming model and language chosen, use an appropriate compiler available in the NVIDIA HPC SDK to compile your application:

• NVIDIA HPC Compilers
  • nvfortran
  • nvc
  • nvc++
• NVIDIA CUDA C/C++ Compiler - nvcc

Run the command module avail nvhpc to find available versions of NVIDIA HPC SDK on the system. The examples shown in this article use nvhpc/23.7. However, you can choose a different version.

% module load nvhpc/23.7

or

% module load nvhpc-nompi/23.7

For building NVIDIA GPU applications with MPI for inter-GPU or inter-node communication, you can use either OpenMPI or HPE MPT:

OpenMPI from nvhpc

Example 1:

module load nvhpc/23.7
mpif90 -o your_application.exe -acc=gpu -gpu=cc80 
     -Minfo=accel your_application.f

Note: The mpif90 line is shown on two lines due to KB formatting issues. It should be a single line.

Example 2:

module load nvhpc/23.7
mpicc -o main.o -c main.c 
nvcc  -o cuda_code.o -c cuda_code.cu 
mpicc -o your_program.exe main.o cuda_code.o -lcudart

Note: The default OpenMPI version in nvhpc/23.7 is 3.1.5.

HPE MPT library

This is recommended for building/running applications when using NAS GPU nodes supported by HPE.

Example 1:

module load nvhpc-nompi/23.7
module load mpi-hpe/mpt
nvfortran -o your_application.exe \
-acc=gpu -gpu=cc80 -Minfo=accel your_application.f -lmpi

Example 2:

module load nvhpc-nompi/23.7
module load mpi-hpe/mpt
nvc  -o main.o -c main.c -lmpi
nvcc -o cuda_code.o -c cuda_code.cu
nvc  -o your_program.exe main.o cuda_code.o -lcudart -lmpi

Instead of explicitly using nvfortran/nvc/nvc++ ... -lmpi in the command line, you can use the mpif90, mpicc, and mpicxx wrappers from HPE MPT. However, it is better to set the following variables before invoking mpif90, mpicc, and mpicxx to ensure nvfortran/nvc/nvc++ are used instead of gfortran/gcc/g++:

setenv MPIF90_F90 nvfotran
mpif90 [options] program.f

setenv MPICC_CC   nvc
mpicc  [options] program.c

setenv MPICXX_CXX nvc++
mpicxx [options] program.cc

Running a CUDA Application

The following environment variables may be useful during runtime:

• MPI_USE_CUDA: If using HPE MPT library for running a CUDA application across multiple GPUs or across multiple nodes, setting this variable will enable CUDA support so that the buffers on the GPUs can be used with sends, receives, collectives, and MPI RMA calls.
• CUDA_VISIBLE_DEVICES: Use this variable to control which GPUs your application uses. This is useful when you want to target a specific GPU or a set of GPUs for debugging or testing.

Sample PBS Script

#PBS -l select=1:ncpus=16:ngpus=4:mem=240GB:model=mil_a100
#PBS -l walltime=8:00:00
## If you request more than 1 chunk (i.e., select=2 or more),
## with resources which are scattered (i.e., not using the same physical nodes)
## and you want exclusive access to the vnodes assigned,
## uncomment the following place statement
##PBS -l place=scatter:excl
#PBS -q gpu_devel

module load nvhpc-nompi/23.7
module load mpi-hpe/mpt

cd $PBS_O_WORKDIR

setenv MPI_USE_CUDA 1

## if you want to run on two specific GPUs, 
## for example, GPU0 and GPU2

setenv CUDA_VISIBLE_DEVICES 0,2
mpiexec -np 2 ./your_application.exe
 
## if you want to run on all four GPUs in the node

setenv CUDA_VISIBLE_DEVICES 0,1,2,3
mpiexec -np 4 ./your_application.exe

References

• Cabeus Configuration Details 
• PBS Job Sharing on GPU Nodes
• Requesting GPU Resources
• Programming for NVIDIA GPUs