Cabeus, named after a lunar crater in the south polar region of the Moon, is a cluster system that supports HPC and machine learning/artificial intelligence (ML/AI) applications that can benefit from the GPU technology.
GPU Hardware Summary
The node statistics are shown in two groups: nodes based on the x86_64 operating system (OS), and those based on the aarch64 OS.
Cabeus x86_64 GPU Nodes
12 racks
128 nodes
8,192 CPU cores + 1,769,472 double-precision GPU cores
Total memory: 106 terabytes (65 TB from CPU host memory + 41 TB from GPU memory)
Theoretical-double precision peak performance: 5.31 Petaflop/s (0.32 Pflop/s from CPUs + 4.99 Pflop/s from GPUs)
Cabeus aarch64 GPU Nodes
23 racks
538 nodes
38,736 CPU cores + 4,545,024 double-precision GPU cores
Each GPU compute node includes one CPU socket as the host and one or four GPU cards as the coprocessors. These compute nodes are grouped by the CPU processor models as listed below:
model=mil_a100
128 nodes total. Each node contains one AMD Milan CPU socket and four NVIDIA Amper A100 GPU cards, where the CPU and GPUs are connected via PCI Express 4.0 bus.
The Milan CPU socket used in the mil_a100 nodes are AMD EPYC 7763.
model=gh200
538 nodes total. Each node contains one NVIDIA GH200 Grace Hopper Superchip, which consists of one Grace CPU (Neoverse V2 core with ARM v9.0-A architecture) and one H100 GPU, connected by a high-bandwidth, memory-coherent NVLink-C2C interconnect.
Cabeus Processor, Memory, and Network Subsystems Statistics
Maximum Double
Precision Floating Point
Operations per cycle per core
16
16
Total CPU Double Precision Flops/Node
2,508 GFlops
3.99 TFlops
Host Memory/Node
512 GB (8 x 64 GB DDR4)
480 GB (LPDDR5X)
GPU
GPU Architecture
Ampere
Hopper
Compute Capability**
cc80
cc90
Device Name
NVIDIA A100-SXM4-80GB
NVIDIA GH200 480GB
Clock Rate (Base/Boost)
1,275 MHz/1,410 MHz
(n/a)/1,980 MHz
# of coprocessors/Node
4
1
# of Streaming
Multiprocessors
(SMs)/coprocessor
108
132
# of Single Precision
CUDA Cores/coprocessor
6,912
16,896
# of Double Precision
CUDA Cores/coprocessor
3,456
8,448
# of Tensor Cores/coprocessor
432
528
Maximum double precision
floating point operations per cycle per CUDA core
2
2
Total GPU Double Precision
Flops/Coprocessor (Base/Boost)
using Double Precision CUDA Cores
8.8 TFlops /9.7 TFlops
33.4 TFlops
Total GPU Double Precision
Flops/Node (Base/Boost)
using Double Precision CUDA Cores
35.2 TFlops/38.8 TFlops
33.4 TFlops
L2 Cache
40 MB
60 MB
Global Memory/coprocessor
80 GB (HBM2e)
96 GB (HBM3)
Memory Clock Rate
1,593 MHz
2,619 MHz
Memory Bus Width
5,120 bits
6,144 bits
Memory Bandwidth
2,039 GB/s
(if using SXM form factor; 1,935 GB/s if using PCI form factor)
4,000 GB/s
GPU-to-GPU Communication
with NVLink 3.0: 600 GB/s
N/A
CPU-to-GPU Communication
PCIe 4.0 with 31.5 GB/s bandwidth for each direction
NVLink-C2C with 900 GB/s bi-directional
Inter-node Network
IB Device on Node
1 HDR 200 Gb 2-port card
+ 1 GbE 4-port card
NDR 200 Gb 1-port
Local Disk
SSD/node***
raw (2 x 1.92 TB SSD drives
+ 2 x 3.84 TB NVMe SSD) or (6 x 2.93 TB NVMe SSD)
960 GB Samsung NVMe M.2
** The compute capability identifies the features supported by the GPU hardware and is used by applications at runtime to determine which hardware features and/or instructions are available on the present GPU. Do not confuse the compute capability number with the CUDA version.
*** Not yet available for public usage, pending configuration decisions.
For CPU info:
cat /proc/cpuinfo
For host memory info:
cat /proc/meminfo
Note: For the gh200 nodes with the unified memory feature,
the total memory shown in the output of /proc/meminfo is the sum
of the CPU memory and the GPU memory.
For GPU info:
/usr/bin/nvidia-smi -q
or load a nvhpc module and run the command pgaccelinfo or nvaccelinfo.
For example, for systems with x86_64 architecture:
module avail nvhpc
----------------- /nasa/modulefiles/toss4 ----------------
nvhpc-nompi/22.3 nvhpc-nompi/23.11 nvhpc/23.5 nvhpc/24.1
nvhpc-nompi/23.5 nvhpc-nompi/24.1 nvhpc/23.7 nvhpc/openmpi-3.1.5
nvhpc-nompi/23.7 nvhpc/22.3 nvhpc/23.11 nvhpc/openmpi-4.0.5
module load nvhpc/23.11
For example, for systems with aarch64 architecture:
module avail nvhpc
module load nvhpc/24.11
pgaccelinfo
or
nvaccelinfo
