The Cabeus cluster comprises two types of NVIDIA GPU nodes, including nodes with x86_64-based CPUs and nodes with aarch64-based CPUs.

Note: You must have a Standard Billing Unit (SBU) allocation specifically for GPUs in order to use the GPU nodes.

Nodes with x86_64-Based CPUs

• 128 nodes each with one Milan CPU and four NVIDIA A100 GPUs (model=mil_a100)

These GPU nodes are served by PBS server pbspl4 and are accessible from the Cabeus front ends, cfe01 and cfe02.

The nodes are sharable by different jobs. Instead of each node being treated as one unit for exclusive access by a single job, each node is logically split into four vnodes.

If you want exclusive access to a node, request the use of all the resources in a node and specify the placement -lplace=scatter:excl. for example:

#PBS -lselect=1:ncpus=128:ngpus=4:mem=480GB:model=mil_a100
#PBS -lplace=scatter:excl
#PBS -q queue_name@pbspl4

where the use of @pbspl4 in the line #PBS -q is optional.

Note: There are 64 physical cores per mil_a100 node. However, PBS sees each mil_a100 node as having 128 logical cores due to hyperthreading enabled. If you specify :ncpus=64, PBS will assign 64 logical cores all from the first 32 physical cores. To get all 64 physical cores assigned to your job, use :ncpus=128.

Nodes with aarch64-based CPU

Cabeus contains 538 nodes with NVIDIA GH200 Superchips (model=gh200).

These nodes are accessible from the Cabeus Grace Hopper front ends, cghfe01 and cghfe02. PBS jobs are served by pbs05a, where user login is not allowed.

The gh200 nodes have only one GPU per node. A node is configured via PBS for exclusive use by a single job, and it is not necessary to explicitly request the amount of resources. For example, the following two PBS resource requests provide you with the same exclusive access to all resources of a whole node:

#PBS -lselect=1:model=gh200
or
#PBS -lselect=1:ncpus=72:ngpus=1:mem=460GB:model=gh200
#PBS -q queue_name@pbs05a

where the use of @pbs05a in the line #PBS -q is optional.

Requesting AMD Milan A100 GPU Nodes (mil_a100)

Each mil_a100 node contains only one socket; each socket is further split into four vnodes where each mil_a100 vnode contains 1 GPU, 16 CPU physical cores, and 128 GB of memory.

The new queue structure utilizes the PBS primetime feature. A “nonprime-time” queue, np_gpu_wide, will run jobs only from 7:00 a.m. Thursday to 7:00 a.m. Saturday (Pacific Time). The gpu_normal and gpu_devel queues will run during both primetime and non-primetime (i.e., any time). You can submit jobs to any of the queues at any time during the week, but the non-prime time jobs will run only during their designated period.

The following PBS queues are enabled for use:

• gpu_normal
  • Maximum wall time: 24 hrs
  • Maximum node count: 32
     
• gpu_devel
  • Maximum wall time: 2 hrs
  • Maximum node count: 2
  • Maximum user jobs (queued or running): 1
  • Priority: 10
     
• np_gpu_wide (non-primetime)
  • Maximum wall time: 12 hrs
  • Maximum node count: 64
  • Minimum node count: 24
  • Maximum user jobs (queued or running): 2
  • Priority: 20

Notes:

• Long running jobs need to implement checkpoint/restart if they cannot fit into the wall-time limit of these queues.
• The maximum amount of resources or certain settings for these queues may change from time to time. To get the current setting, run the command qstat -fQ queue_name. Note that the resources_max.nodect value refers to the number of physical nodes allowed per job.

In your job request, specify the number of CPU cores, the number of GPUs, and the amount of host memory you want for your job. The following example shows how to request exclusive usage of a whole node:

#PBS -lselect=1:ncpus=128:ngpus=4:mem=480GB:model=mil_a100
#PBS -lplace=scatter:excl

If ncpus, ngpus, or mem is not specified, the default values (ncpus=1, ngpus=1, mem=32GB) will be used.

Requesting NVIDIA GH200 GPU Nodes (gh200)

To run on the GH200 nodes, submit your PBS interactive or batch job from one of the two Grace Hopper front ends, cghfe01 or cghfe02. Use one of the following queues:

• gpu_normal
  • Maximum wall time: 24 hrs
  • Priority: Default (0)
     
• gpu_devel
  • Maximum wall time: 2 hrs
  • Maximum node count: 2
  • Maximum user jobs (queued or running): 1
  • Priority: 10
     
• gpu_wide
  • Maximum wall time: 12 hrs
  • Priority: 20
  • Wide is considered 85 or more nodes requested for a job.
     
• gpu_long
  • Maximum wall time: 120 hrs
  • Maximum node count: 8
  • Maximum user jobs (queued or running): 2
  • Priority: Default (0)

Note: Since each gh200 node is set up to not allow job sharing, the use of PBS placement such as ‑lplace=scatter:excl is not required.

Charging for Use of the GPU Nodes

A separate SBU allocation exclusively for the GPU nodes is required. For information on requesting SBU allocation for the GPU nodes, see this article.

The SBU rates for using 1 GPU node for 1 hour are:

• mil_a100 with four A100s: 37.86
• gh200 with one H100: 18.34

Note: For the GPU nodes with aarch64-based CPU (model=gh200), your job will be charged for the SBU cost of a 1 whole node regardless of how the resources are requested, since these nodes are not set up for job sharing (each job is allocated with all of the resources in a whole node).

For the GPU nodes with x86_64 based CPUs, since a node is sharable by multiple jobs, your job may be charged for the SBU cost of 1 whole node or a fraction of the node depending on the resource request and the place statement in your PBS job. In addition, charging is based on the number of GPUs allocated to or excluded from use by other jobs, with no consideration of CPUs and memory. Below are three examples.

Example 1

This resource request will be charged for one-quarter of the mil_a100 SBU rate of 37.86, because only 1 GPU will be allocated to the job. In addition, it allows the unused CPU, GPU, and memory in the node to be used by other jobs:

#PBS -lselect=1:ncpus=1:ngpus=1:mem=60GB:model=mil_a100
#PBS -lplace=scatter:shared

Example 2

This job will be charged for 2 times the SBU rate of 37.86, because all four GPUs in each of the two mil_a100 nodes will be allocated and no other jobs will be able to share the resources:

#PBS -lselect=2:ncpus=1:ngpus=4:mem=60GB:model=mil_a100
#PBS -lplace=scatter:excl  

Example 3

This job will be charged for one half (2 x 0.25) the SBU rate of 37.86, because only 1 GPU each from 2 different nodes will be allocated and the remaining 3 GPUs in each node can be used by other jobs:

#PBS -lselect=2:ncpus=1:ngpus=1:mem=60GB:model=mil_a100
#PBS -lplace=scatter:excl

Checking Job Status

To check the status for all of your jobs submitted to a particular queue, run the qstat command. The following examples use the queue gpu_normal served by the pbspl4 PBS server. For jobs submitted to queues served by the pbs05a PBS server, change the front end from cfe to cghfe01 or cghfe02, the PBS server from pbspl4 to pbs05a, and use an appropriate queue name.

cfe% qstat gpu_normal@pbspl4 -u your_username

You can also check the status of a single job. For example, to check the status of job 140101 on pbspbl4:

cfe% qstat 140101.pbspl4.nas.nasa.gov