Cabeus includes 538 GPU nodes based on the NVIDIA GH200 Grace Hopper Superchip.

To help you prepare for running jobs on the Grace Hopper GPU nodes, this short user guide includes information on the general configuration of the nodes, compiling your code, and running PBS jobs.

Overview of Cabeus Nodes with NVIDIA Grace Hopper Superchip

The nodes are accessed via two front ends:

• cghfe01
• cghfe02

The hostnames for the GH200 compute nodes are as follows:

• cgh[1-3,6-15,17-19,21-23]n[01-26]
• cgh[04,20]n[01-12]
• cgh16n[01-20]

The PBS resource model name is gh200.

Each of the front ends and compute nodes contains a single GH200 superchip, which is shown in this diagram published by NVIDIA:

As shown in the diagram, the GH200 superchip includes:

• One Grace CPU containing 72 Arm Neoverse V2 physical cores (maximum 3.465 GHz on the front ends; slight variation on compute nodes but should be all above 3.4 GHz; use the lscpu command to check) and up to 480 GB Low-Power Double Data Rate 5X (LPDDR5X) host memory with a memory bandwidth of up to 500 GB/s.

  Note: The memory bandwidth varies with memory capacity: 384 GB/s with a capacity of 480 GB as in all NAS GH200 nodes, and 512 GB/s with a capacity of 120 GB or 240 GB.

  The Arm Neoverse V2 CPU uses the Armv9.0-A architecture which comes with four 128-bit Single Instruction Multiple Data (SIMD) units that support Scalable Vector Extension 2 (SVE2) instructions.
• One H100 (Hopper) GPU containing 132 Streaming Multiprocessors (SMs) with a clock rate of 1.980 GHz and 96 GB High Bandwidth Memory 3 (HBM3) device memory with a memory bandwidth of up to 4000 GB/s. Each SM contains 128 single precision CUDA cores (total is 16,896 per H100 GPU), 64 double precision CUDA cores (total is 8,448 per H100 GPU), and 4 Tensor cores (total is 528 per H100 GPU).
• A high-bandwidth, memory-coherent NVIDIA NVLink-C2C (chip-2-chip) interconnect that connects the CPU and GPU with a bidirectional bandwidth of 900 GB/s.
  
  NVLink-C2C enables applications to oversubscribe the Hopper GPU's memory and directly utilize the Grace CPU's memory at high bandwidth. The Grace CPU can also directly access the GPU memory. The memory coherency enables programming with a unified programming model.
• 4x (4 links) PCIe 5.0 x16 (16 lanes) with a total bi-directional bandwidth of 512 GB/s.
  
  The GH200 nodes at NAS use the NVIDIA MGX platform design where the nodes are interconnected with PCIe and host channel adapter (HCA) on each node and NVIDIA's internode InfiniBand networking solution. Specifically, InfiniBand Next-Generation Data Rate (NDR) HCAs, switches, and cables are utilized in a fat tree topology.
  
  Note: The 18x NVLink 4 network shown on the right side of the diagram above does not apply to the NVIDIA MGX platform. It is used in the NVIDIA DGX platform for connecting up to 256 GPUs together.

For more information about the Grace Hopper superchip, see NVIDIA's GH200 architecture whitepaper.

Requesting GPU Allocations

A Standard Billing Unit (SBU) allocation specifically for the GPUs is required to use all GPU resources, including the GH200 compute nodes. The SBU rate for the GH200 nodes is 18.34.

Accessing the Cabeus Grace Hopper Front Ends and Compute Nodes

The GH200 front end nodes (cghfe01 and cghfe02) are for building, testing your software dependencies, and running short tests of your executables.

Note: Termination of long-running CPU and GPU processes may be enforced in the future.

You can access the GH200 front ends from your local desktop using one of the following methods:

One-Step Connection With SSH passthrough

If you normally connect to NAS front-ends using SSH passthrough, you can modify your existing .ssh/config file on your local host to add cghfe01 and cghfe02. If you have been using a configuration file based on the NAS config template, you can simply replace this line:

Host *.nas.nasa.gov lou lou? ?fe ?fe? ?fe?? *-last pbspl?

with the following line:

Host *.nas.nasa.gov lou lou? *fe* *-last pbs*

where *fe* matches any NAS front-end system. Then, run:

your_localhost% ssh cghfe01 (or ssh cghfe02)

Two-Step Connection Without SSH Passthrough

To access the front ends without SSH passthrough, run:

your_localhost% ssh sfe6 (or ssh sfe7, ssh sfe8)
sfe% ssh cghfe01  (or ssh cghfe02) 
Password: xxxxxxxxxxx
cghfe01%

The gh200 compute nodes are for running long testing or production jobs. They are accessible only through PBS jobs submitted from the cghfe01 or cghfe02 front ends. The PBS server for the GH200 nodes is pbs05a (users are not allowed to SSH into pbs05a).

You can access your filesystems from the GH200 front-ends and compute nodes, including: $HOME, Lustre /nobackup, /nobackupnfs1, /nasa, /nasa2, and /vast_swbuild.

Operating Systems

The operating system on all GH200-based front ends and compute nodes is TOSS 5 with Red Hat Enterprise Linux 9.8. The Linux kernel is Release 5.x for the Arm architecture 64-bit execution state (aarch64).

% cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.8 (Plow)"

% uname -r
5.14.0-687.39.1.1toss.t5.aarch64

Software Modules

For AI/ML Applications

If you run AI/ML applications: the miniconda3/gh2 module in the /swbuild/analytix/tools directory is the only one that will work with the GH200 nodes:

cghfe01% module use -a /swbuild/analytix/tools/modulefiles
cghfe01% module load miniconda3/gh2

For more information, including the conda environments that can be used with the Grace Hopper nodes, see Using Conda Environments on NAS Systems.

The procedure to access JupyterLab on the GH200 system is the same as those outlined in Secure Setup for Using Jupyter Notebook on NAS Systems and Using Jupyter Notebook for Machine Learning Development on NAS Systems, except on the local system the user needs to proxyjump through a Grace Hopper front end. For example:

your_local_system% ssh -l username -o "StrictHostKeyChecking ask" \
-L 8080:/tmp/user.sock -o ProxyJump=sfe,cghfe01

For HPC Applications

You can find HPC software modules for the Arm-based GH200 nodes in the /nasa/arm directory. The current default $MODULEPATH includes /nasa/arm/modulefiles/toss5, where you can find many of the modules mentioned in this article.

cghfe01% echo $MODULEPATH
/etc/scl/modulefiles:/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/ \
modulefiles:/nasa/arm/modulefiles/toss5

Note: The command line above is too long to be formatted as one line, so it is broken with a backslash (\).

Some of the available software packages are mentioned in the Programming and Compiling for Grace Hopper Nodes and Running a GPU Application sections. Be aware that many nvhpc or nvhpc-xxx modules (packaged and pre-built by NVIDIA) contain multiple components (such as CUDA, compiler, math, MPI, .., and so on.). If you want to use only one of the components, you can try other modules, for example, those with module names such as cuda, hpcx, openmpi, and so on.

Programming and Compiling for Grace Hopper Nodes

In addition to the information in this section, read Programming for NVIDIA GPUs and the references listed there for information on the recommended programming models for developing applications suitable for running on the NVIDIA GPUs. Some of the most commonly used software packages needed with these programming languages are CUDA toolkits, compilers, and math and MPI libraries.

CUDA Toolkit

You can find the CUDA version and its associated driver installed on the GH200 front ends and compute nodes by running the nvidia-smi command. Currently, it is CUDA version 13.3 and driver is 610.57.04.

Since the CUDA driver is backward compatible, an application compiled with an older CUDA Toolkit version can run with a newer driver release. For example, you can build your GPU application with CUDA Toolkit 12.6, and run it on a system with CUDA 13.3. See CUDA Toolkit and CUDA Driver for more information.

The following CUDA Toolkits are available for use on the GH200 nodes:

• CUDA 12.6

  module load nvhpc/25.1 (or module load nvhpc-nompi/25.1)
  nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2024 NVIDIA Corporation
  Built on Wed_Oct_30_00:08:18_PDT_2024
  Cuda compilation tools, release 12.6, V12.6.85
  Build cuda_12.6.r12.6/compiler.35059454_0
  
  OR
  
  module load nvhpc/24.11 (or module load nvhpc-nompi/24.11)
  nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2024 NVIDIA Corporation
  Built on Thu_Sep_12_02:27:38_PDT_2024
  Cuda compilation tools, release 12.6, V12.6.77
  Build cuda_12.6.r12.6/compiler.34841621_0

• CUDA 12.3

  module load nvhpc/24.3 (or nvhpc-nompi/24.3 or nvhpc-openmpi3/24.3)
  nvcc --version
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2023 NVIDIA Corporation
  Built on Fri_Nov__3_17:22:02_PDT_2023
  Cuda compilation tools, release 12.3, V12.3.103
  Build cuda_12.3.r12.3/compiler.33492891_0

Compilers

There are several compiler options for building your applications on the GH200 front ends. With any of these compilers, make sure your application builds, runs successfully, and validates correctly.

NVIDIA Compilers: nvc, nvc++, nvfortran, nvcc

To use an NVIDIA compiler, load one of the nvhpc modulefiles on either the cghfe01 or cghfe02 front end or on the GH200 compute nodes. Below are examples of a few possible choices:

• For the 25.1 version of the NVIDIA compilers, use:

  % module load nvhpc/25.1
  or
  % module load nvhpc-nompi/25.1

• For the 24.11 version of the NVIDIA compilers, use:

  % module load nvhpc/24.11
  or
  % module load nvhpc-nompi/24.11

• For the 24.3 version of the NVIDIA compilers, use:

  % module load nvhpc/24.3
  or
  % module load nvhpc-nompi/24.3

The NVIDIA compilers include the HPC compilers and the CUDA C/C++ compiler:

• NVIDIA HPC Compilers: nvc, nvc++, nvfortran
  
  These compilers can be used to compile pure CPU applications or GPU applications written in C/C++ and Fortran language standards (through the -stdpar[=gpu,multicore] flag); OpenACC (-acc[=gpu,multicore] flag); OpenMP (-mp[=gpu,multicore] flag), and CUDA Fortran. The gpu and multicore options let you choose whether to compile the application to run on the GPUs or on the multicore CPUs.
  
  Use the -tp=native flag or the -mcpu=native flag when compilation and execution occur on the same CPU processor (in this case, the Grace CPU) and use the -gpu=cc90 flag (i.e., compute capability 9.0) to target the Hopper GPU.
   
• NVIDIA CUDA C/C++ Compiler: nvcc
  
  The nvcc compiler is used to build CUDA C/C++ applications to run on the GPUs. This compiler splits the source code into device code and host code. The device code is compiled by nvcc. nvcc sends the host code to a C/C++ compiler on the host for compilation—for example, the gcc and g++ GNU compilers in the /usr/bin directory. You can choose a different host compiler—for example, a different version of GNU compiler—through the nvcc flag -ccbin=/path_to_compiler or --compiler-bindir=/path_to_compiler.

For more information, see Compiling GPU Applications for Different Generations of NVIDIA GPUs at NAS.

GNU Compiler Collection: gcc, g++, gfortran

• GNU compilers at the system location:

  cghfe01% which gcc
  /usr/bin/gcc
  cghfe01% gcc --version
  gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-14)
  
  cghfe01% which gfortran
  /usr/bin/gfortran
  cghfe01% gfortran --version
  GNU Fortran (GCC) 11.5.0 20240719 (Red Hat 11.5.0-14)

  These compilers can only be used for building CPU-only applications, as they were not built with offloading support.

For all of the above compilers, NVIDIA recommends:

• Use the -mcpu=native flag instead of the -march=xxx or -mtune=xxx flags to optimize for the native CPU and use its features.
• Depending on the char sign type you assume for your application. you may need to use one of these C language flags: -fsigned-char, -funsighed-char, or -fnosigned-char.
  Note: For NVIDIA HPC compilers (nvc, nvc++, nvfortran), the default char sign type is -fsigned-char on x86_64 systems and -funsighed-char on aarch64 systems. For the Arm Compiler for Linux, the default is -fno-signed-char.

For the GNU compiler and/or Arm compiler:

• If using -Ofast results in unacceptable fast math optimizations, use -O3 -ffp-contract=fast. For even more accuracy, use -ffp-contract=off to disable floating point operation contraction used in fused multiply-add.
• Use -flto to enable link-time optimization, which can be significant.

For the GNU compiler, you can use -fsanitize=address -fsanitize=undefined to help debugging memory issues that might appear on aarch64 systems (but are hidden on x86 systems due to differences on how compiler generate codes and layout data).

Math Libraries

There are two sets of math libraries. One targets the GPUs and the other targets the Grace CPUs.

• CUDA libraries targeting the Hopper GPUs:
  This set of libraries includes: cuBLAS, cuFFT, cuRAND, cuSOLVER, cuSPARSE, and cuTENSOR. They are located in the following directories:
  • If using module nvhpc/25.1:
    /nasa/arm/software/nvhpc/25.1/Linux_aarch64/25.1/math_libs/lib64
  • If using module nvhpc/24.11: /nasa/arm/software/nvhpc/24.11/Linux_aarch64/24.11/math_libs/lib64
  • If using module nvhpc/24.3: /nasa/arm/software/nvhpc/24.3/Linux_aarch64/24.3/math_libs/lib64
  When using the NVIDIA HPC compilers (nvc, nvc++, and nvfortran), these libraries can be linked with the flag -cudalib[=cublas|cufft|...].
   
• NVIDIA Performance Libraries (NVPL) targeting the Grace CPUs:
  NVPL is a set of optimized math libraries (BLAS, FFT, LAPACK, RAND, SCALAPACK, SPARSE, and TENSOR) that are used for the Grace CPUs. The NVPL set is available with NVIDIA HPC SDK version 24.3 (and later) in the /nasa/arm/.../math_libs/nvpl/libs directory.
  When using the NVIDIA HPC compilers (nvc, nvc++, and nvfortran), these libraries (with the exception of SCALAPCK) can be linked with the flag -Mnvpl[=blas,fft,...]. To link with NVPL SCALAPACK, use both -Mscalapack and -Mnvpl.
  If using the GNU compiler for linking with NVPL, use the -I, -L, and -lnvpl_.... options explicitly.

MPI Libraries

The GH200 nodes are not supported by HPE, and therefore HPE MPT library is not available on these nodes.

There are many modules containing OpenMPI. If you are not sure which one to use, start with one of these OpenMPI modules. Read Using OpenMPI on NAS Systems to learn how to use OpenMPI.

OpenMPI Version 4.1.7

• nvhpc/25.1: provided by NVIDIA, and built with gcc, g++, nvfortran, HPCX 2.21, and CUDA 12.6.
• nvhpc/24.11: provided by NVIDIA, and built with gcc, g++, nvfortran, HPCX 2.20, and CUDA 12.5.
• nvhpc/24.3: provided by NVIDIA, and built with gcc, g++, nvfortran, HPCX 2.17.1, and CUDA 12.2.
• hpcx/2.21: open source version of precompiled HPCX. The OpenMPI provided in this distribution was built with gcc, g++, gfortran, HPCX 2.21, and CUDA 12.6.
• openmpi/4.1.7-cuda12.6-gnu: built by NAS staff with GNU compiler, UCX 1.17, and CUDA 12.6.
  Loading openmpi/4.1.7-cuda12.6-gnu automatically loads nvhpc-nompi/24.11, which provides access to CUDA 12.6.
• openmpi/4.1.7-cuda12.6-nv: built by NAS staff with nvc, nvc++, nvfortran, UCX 1.17, and CUDA 12.6.
  Loading openmpi/4.1.7-cuda12.6-nv automatically loads nvhpc-nompi/24.11 which provides access to NVIDIA HPC compilers and CUDA 12.6.

OpenMPI Version 3.1.5

• nvhpc-openmpi3/24.3: provided by NVIDIA, and built with nvc, nvc++, nvfortran, CUDA 11.0, and without UCX.

mpiexec -np .. -x UCX_PROTO_ENABLE=n -x UCX_GVA_ENABLE=n ...

UCX_IB_MLX5_DEVX=no

Running a GPU Application

PBS jobs running on the GH200 nodes are served by a dedicated PBS server, pbs05a, running with PBS version 2024.1.4.

A GH200 compute node (72 CPU cores; 1 GPU; all host and device memory) cannot be divided or shared between PBS jobs or between different chunks of a PBS job (for example, -lselect=N means N chunks). The GH200 node model name is gh200. In your PBS resource request, you must use :model=gh200. You do not need to use -lplace=scatter:excl.

To run on the GH200 nodes, submit your PBS interactive or batch job from one of the two Grace Hopper front ends, cghfe01 or cghfe02. Use one of the following queues:

• gpu_normal: default priority (0), maximum walltime 24 hours.
• gpu_devel: priority 10, maximum walltime 2 hours, number of nodes 2.
• gpu_wide: priority 20, maximum walltime 12 hours (wide is considered 85+ nodes requested for the job).
• gpu_long: default priority (0); maximum walltime 120 hours; number of nodes 8; number of running jobs 2.

Note: The available queues will change from time to time. Use the command qstat -Q and qstat -fQ to find what queues have been created and enabled to be used by your group.

See the following sample PBS script.

Sample PBS Script for Running Jobs on Grace Hopper Nodes

#!/bin/bash
#PBS -lselect=2:ncpus=72:ngpus=1:model=gh200
#PBS -q name_of_queue
 
module load nvhpc/24.11

# for running 144 MPI ranks on the CPU cores across two nodes
# with 72 MPI ranks per node
# if necessary, also add -x LD_LIBRARY_PATH
mpiexec -np 144 -N 72 --machinefile $PBS_NODEFILE ./a_CPU_executable

# for running 2 MPI ranks on two GPUs across two nodes
# with 1 MPI rank per node
# if necessary, also add -x LD_LIBRARY_PATH
mpiexec -np 2 -N 1 --machinefile $PBS_NODEFILE ./a_GPU_executable

After job submission, you should see a JOBID similar to xxx.pbs0a5.gh.nas.nasa.gov.

Note: The versions of qstat on newer NAS systems (including Athena and Cabeus GH200 nodes) and older NAS systems (including Electra, Aitken, and Endeavour, and Cabeus V100 and A100 nodes) are different. To get the same look and feel of qstat on the newer systems as the older systems, it is recommended to use a NAS provided script called nas_qstat, available under /usr/local/bin on athfe[01-04] and cghfe[01-02]. In addition, the script showq under /u/hsp/bin can be used to visually see all jobs that are running or pending and their status, along with node availability at the bottom of the output.

References

• NVIDIA Grace Hopper Superchip Architecture White Paper
• NVIDIA Grace Performance Tuning Guide